* Re: Update-Accelerator 3.0
[not found] <SNT136-W471D0DEDB59EA0B88832CEC5E40@phx.gbl>
@ 2013-03-06 14:09 ` Michael Tremer
0 siblings, 0 replies; 2+ messages in thread
From: Michael Tremer @ 2013-03-06 14:09 UTC (permalink / raw)
To: development
[-- Attachment #1: Type: text/plain, Size: 4161 bytes --]
Hello,
On Wed, 2013-03-06 at 16:57 +0800, Fajar Ramadhan wrote:
> Hello there, replying inline
>
> > Any other ideas?
>
> Hyper Cache
That's a possibility. I didn't know that anyone is still using the word
hyper :D
> > >> Michael from IPFire.org told me that you may have some
> requirements or
> > >> ideas for an improved update accelerator
>
> Firstly, this idea is not part of update accelerator thingie.
Well, we are thinking about a rewrite, so every idea is welcome. Nobody
promises that it will be implemented, but in the process of searching
for the real purpose of the update accelerator, feel free to write
anything on your mind if you think it is worth considering.
> > >> cause we plan to extend the
> > >> current version (at this point it look like a complete rewrite
> o_O)
>
> complete rewrite, maybe :)
?
> My idea is basically out from squid 2.7 abilities to cache dynamic
> contents using built-in storeurl feature.
> http://www.squid-cache.org/Doc/config/storeurl_rewrite_program/
As we are looking into the (far) future, we cannot possibly stick to an
old version of squid. Even the currently in IPFire 2 running version 3.1
is "old" right now.
Maybe it is also a good idea to design this without considering squid as
the default thing to work with it. It should be possible to drop squid
and use an other proxy server - although I really don't have plans for
that right now, because squid is the best proxy server one can have.
> Wiki for example how to use storeurl
> http://wiki.squid-cache.org/Features/StoreUrlRewrite
>
> We already know that squid 2.7 is already obsolete - but this feature
> was extremely useful for slow internet users (just like me in
> Indonesia, where bandwidth is expensive). Built-in storeurl feature
> inside squid 2.7 has ability to cache or manipulating caching for
> dynamically hosted contents (dynamic contents). Example for Facebook's
> CDN :
It is interesting that this has not been ported to squid 3.x.
Apparently, the reason is that this the implementation was poorly
written and so people thought about replacing it entirely. It also looks
like there are not many users of this feature.
> If squid already cache one of this picture then all same pictures from
> hprofile-ak-prn1, hprofile-ak-prn2, hprofile-ak-prn3 .....
> hprofile-ak-prnX will result cache hit - squid not necessary to fetch
> same content from different cdn urls, since its already in cache and
> request got rewritten by storeurl. All contents from Facebook such as
> javascript, css, image, even sound and videos will have very high
> chance to get hits from squid.
Looking at your user data your provided further below, the important
stuff to cache is big files. That's not only video and all sorts of
downloads. Nowadays javascript code of sites like Facebook is of the
size of one or two megabytes*.
* Didn't check. Read this somewhere, some time ago.
What I get from this is that we should design the rewrite to literally
cache anything.
A technical question from me: Why cannot we use the internal cache of
squid to do so, but code our own caching proxy that is actually queried
by the real caching proxy? I think even with a very fast implementation,
squid will always be much faster.
> This method works on almost all web that serving dynamic contents for
> its visitors : Youtube videos (all resolutions) , blogger.com
> contents, online games patch files, google maps, ads, imeem, etc. This
> is something that you cannot done with squid 3.x.
This cannot be done with squid 3 AT THE MOMENT.
> Another approach to make it work on squid 3 is using ICAP - I'm not
> familiar with this one since I never used it. You can see some
> reference about ICAP to cache dynamic contents here (for me it seems
> difficult to do it) :
> http://www.squid-cache.org/mail-archive/squid-users/201206/0074.html
As pointed out earlier, I like ICAP. The protocol has a lot of
advantages and makes us independent from squid (not to replace it, but
being not dependent on a certain version - they all talk ICAP).
Can someone find out if somebody already implemented this kind of thing?
Terima kasih,
-Michael
^ permalink raw reply [flat|nested] 2+ messages in thread
* Re: Update-Accelerator 3.0
[not found] <CAEpw26UAAVxm3r2CYbJ279GdKmdTebnvjXx7sYP7LhbOD7d_Bw@mail.gmail.com>
@ 2013-03-06 19:44 ` Michael Tremer
0 siblings, 0 replies; 2+ messages in thread
From: Michael Tremer @ 2013-03-06 19:44 UTC (permalink / raw)
To: development
[-- Attachment #1: Type: text/plain, Size: 7288 bytes --]
Hi,
it is great that you take part in the discussion.
On Wed, 2013-03-06 at 20:13 +0100, Jan Behrens wrote:
> I agree to Fajar's intention!
> We need a way more dynamic content caching capability.
No, I think that is exactly the wrong way. Dynamic content should not be
cached because it is _dynamic_. It's very unlikely that someone else
will get the same response to a request.
The only thing that makes sense it to cache _static_ content like
avatars, videos, (big) pictures. But actually this is the proxy's task,
which apparently does not work very well for some things.
> I think the way to go is an written Addon for squid which does the
> needed work.
> Here is a point to start reading about:
> http://wiki.squid-cache.org/Features/AddonHelpers#HTTP_Redirection
What work? Re-implementing an extra cache is not a good idea. At least
not for small files.
> In the company I work in, it is as follows:
> The most traffic is secured (HTTPS).
> Just think about the hosting providers like Dropbox, Google-Drive, and
> so on. There probably are files which are needed by many people and
> then downloaded by them, and you can't legally cache them. That's a
> shame.
Files transferred over HTTPS cannot be cached by the proxy. This has no
legal reasons. It is just technically impossible.
>
> 2013/3/6 Michael Tremer <michael.tremer(a)ipfire.org>
> Hello,
>
> On Wed, 2013-03-06 at 16:57 +0800, Fajar Ramadhan wrote:
> > Hello there, replying inline
> >
> > > Any other ideas?
> >
> > Hyper Cache
>
>
> That's a possibility. I didn't know that anyone is still using
> the word
> hyper :D
>
> > > >> Michael from IPFire.org told me that you may have some
> > requirements or
> > > >> ideas for an improved update accelerator
> >
> > Firstly, this idea is not part of update accelerator
> thingie.
>
>
> Well, we are thinking about a rewrite, so every idea is
> welcome. Nobody
> promises that it will be implemented, but in the process of
> searching
> for the real purpose of the update accelerator, feel free to
> write
> anything on your mind if you think it is worth considering.
>
> > > >> cause we plan to extend the
> > > >> current version (at this point it look like a complete
> rewrite
> > o_O)
> >
> > complete rewrite, maybe :)
>
> ?
>
> > My idea is basically out from squid 2.7 abilities to cache
> dynamic
> > contents using built-in storeurl feature.
> >
> http://www.squid-cache.org/Doc/config/storeurl_rewrite_program/
>
>
> As we are looking into the (far) future, we cannot possibly
> stick to an
> old version of squid. Even the currently in IPFire 2 running
> version 3.1
> is "old" right now.
>
> Maybe it is also a good idea to design this without
> considering squid as
> the default thing to work with it. It should be possible to
> drop squid
> and use an other proxy server - although I really don't have
> plans for
> that right now, because squid is the best proxy server one can
> have.
>
> > Wiki for example how to use storeurl
> > http://wiki.squid-cache.org/Features/StoreUrlRewrite
> >
> > We already know that squid 2.7 is already obsolete - but
> this feature
> > was extremely useful for slow internet users (just like me
> in
> > Indonesia, where bandwidth is expensive). Built-in storeurl
> feature
> > inside squid 2.7 has ability to cache or manipulating
> caching for
> > dynamically hosted contents (dynamic contents). Example for
> Facebook's
> > CDN :
>
>
> It is interesting that this has not been ported to squid 3.x.
> Apparently, the reason is that this the implementation was
> poorly
> written and so people thought about replacing it entirely. It
> also looks
> like there are not many users of this feature.
>
> > If squid already cache one of this picture then all same
> pictures from
> > hprofile-ak-prn1, hprofile-ak-prn2, hprofile-ak-prn3 .....
> > hprofile-ak-prnX will result cache hit - squid not necessary
> to fetch
> > same content from different cdn urls, since its already in
> cache and
> > request got rewritten by storeurl. All contents from
> Facebook such as
> > javascript, css, image, even sound and videos will have very
> high
> > chance to get hits from squid.
>
>
> Looking at your user data your provided further below, the
> important
> stuff to cache is big files. That's not only video and all
> sorts of
> downloads. Nowadays javascript code of sites like Facebook is
> of the
> size of one or two megabytes*.
>
> * Didn't check. Read this somewhere, some time ago.
>
> What I get from this is that we should design the rewrite to
> literally
> cache anything.
>
> A technical question from me: Why cannot we use the internal
> cache of
> squid to do so, but code our own caching proxy that is
> actually queried
> by the real caching proxy? I think even with a very fast
> implementation,
> squid will always be much faster.
>
> > This method works on almost all web that serving dynamic
> contents for
> > its visitors : Youtube videos (all resolutions) ,
> blogger.com
> > contents, online games patch files, google maps, ads, imeem,
> etc. This
> > is something that you cannot done with squid 3.x.
>
>
> This cannot be done with squid 3 AT THE MOMENT.
>
> > Another approach to make it work on squid 3 is using ICAP -
> I'm not
> > familiar with this one since I never used it. You can see
> some
> > reference about ICAP to cache dynamic contents here (for me
> it seems
> > difficult to do it) :
> >
> http://www.squid-cache.org/mail-archive/squid-users/201206/0074.html
>
>
> As pointed out earlier, I like ICAP. The protocol has a lot of
> advantages and makes us independent from squid (not to replace
> it, but
> being not dependent on a certain version - they all talk
> ICAP).
>
> Can someone find out if somebody already implemented this kind
> of thing?
>
> Terima kasih,
> -Michael
>
>
> _______________________________________________
> Development mailing list
> Development(a)lists.ipfire.org
> http://lists.ipfire.org/mailman/listinfo/development
>
>
>
^ permalink raw reply [flat|nested] 2+ messages in thread
end of thread, other threads:[~2013-03-06 19:44 UTC | newest]
Thread overview: 2+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
[not found] <SNT136-W471D0DEDB59EA0B88832CEC5E40@phx.gbl>
2013-03-06 14:09 ` Update-Accelerator 3.0 Michael Tremer
[not found] <CAEpw26UAAVxm3r2CYbJ279GdKmdTebnvjXx7sYP7LhbOD7d_Bw@mail.gmail.com>
2013-03-06 19:44 ` Michael Tremer
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox