GeTor: Provide Tor Browser Bundle download file from Tor2web static URL

[fpietrosanti]

This ticket is to provide Tor Browser Bundle download file from Tor2web static URL, such as /GetTor .

Nowdays downloading Tor is problematic, either in terms of usability, either in terms of accessibility of torproject.org from countries that apply censorship.

For that purpose we are going to implement the GetTor functionality on Tor2web, that will directly serve the latest stable TorBrowserBundle for the user language.

The TorBrowser will be served only for Windows and Mac OS X, where Desktop users are.

If the user language is not available from the ones supported by Tor, it shall default to English.

The /GetTor should detect if the user is already using Tor Browser Bundle and, in such care, redirect him to the TorProject download page.

The /GetTor should detect if the user is on iPhone and, in such case, redirect him to the OnionBrowser AppStore page.

The /GetTor should detect if the user is on Adroid and, in such case, redirect him to the OrBot Google Store page.

If the operating system is unknown, it shall provide an html or text file, saying that it cannot detect the user operating system, suggesting to manually access the torproject.org download web page (edge-case are not managed).

The latests version must be known/detected from https://www.torproject.org/projects/torbrowser/RecommendedTBBVersions .

The file should be served to the end-user in a "streaming way", by avoiding buffering the download in Tor2web, as it would delay the Download window on the end-user side.

Whenever a specific version has been already downloaded, a cached copy must be kept on a Tor2web cache directory.

[fpietrosanti]

This download link will be used also and specifically from within the GlobaLeaks Privacy Badge, in a project where the end-user cannot access torproject.org to download Tor Browser Bundle.

[ilv]

Hi. this looks like a good idea, but it is not clear to me how TB would be served from tor2web. For what I understand, tor2web act as a proxy, so from where would you be getting these files?

[evilaliv3]

@ileiva it acts as proxy but it can also serve specific resources under precise paths.

the idea is to serve original binaries from torproject.org but recongnizing the user language and user OS. this may help also widespreading TBB in conditions where domains like the torproject.org are censored.

[fpietrosanti]

@ileiva Are you going to support such development? :-)

[ilv]

@evilaliv3 Ok, thanks for the clarification.
@fpietrosanti I would gladly help in what I can. I'm currently the lead developer at GetTor, and I'm interested in ideas that go in the same direction. My only concern with this is how easy/hard would be to be considered as an official distribution channel by Tor Project (I mean, like a trusted source/official mirror to download Tor Browser). I think that clearly specifying what are the benefits of doing this would be very helpful.

[fpietrosanti]

@ilv Well, i think that it shall not be considered an official distribution channel by Tor Project.

That's mainly because Tor2web software is run by a plurality of subjects, some more trusted, some less trusted, some unknown, on very different domains.
For that reason, a Tor2web server shall be considered an untrusted source by default.

Even if Tor2web.org domain/cert has been pinned in Chrome/HTTPS Everywhere since several years and it's a service run by people that have a certain level of trust within the Tor Community, i would argue that's not the right channel/path to officially distribute Tor.

That feature request comes as a need for a project we're working with Amnesty International, where:

• their sources cannot download Tor from torproject.org being censored
• their sources lacks basic IT skills, so they will knew about Tor from a custom/highly-usable amnesty branded, privately reachable download instruction page

The general benefit of distributing TBB from Tor2web are:

• Enable download from countries where "torproject.org" is blocked
• Provide "plausible deniability" within the respect of having downloaded TBB (you may be accessing your favorite onion porn)
• Enable third party helper organizations to create "Private Distribution Channels" (ie: instruction web pages) that are focused on specific user-community, in specific language

[ilv]

Great, thanks for the info.

Regarding the code, I think it shouldn't be hard to accomplish. I understand what needs to be done, but as I'm not very familiar with tor2web yet, I'm still not sure if this should be added to an existing module or if I should create a separate script for it. Any guidance on what part of the code should I study or examples of other static URL handlers would be appreciated.

[fpietrosanti]

@ilv super! The code masters of Tor2web that can give you best insight to dig into the code are @evilaliv3 @virgil @hellais ! If you want to join irc channel we're at #tor2web on irc.oftc.net :)

[evilaliv3]

@ilv for what concern working on this ticket i think you have simply to concentrate on the feature and not in the t2w integration. that would be really easy.

if you want i think the best way to implement this feature is to proceed by writing a simple function that given some infos (the user User Agent is the only one meaningful i think) returns the official TBB download url to relocate the user on. to implement this probably you would require an automatically fetching of some urls from TBB; expect this data as static file on the filesystem and i will take care of fetch them for you.

what do you think?

[ilv]

Ok, thanks for the tip @evilaliv3. I think I found where the integration should be, but I just haven't had the time to sit and do some code. In any case, I will concentrate on the feature for now, as you mention. About the fetching, if you have the time to do it, it would be great. Otherwise, I'm willing to do it (checking the RecommendedTBBVersions url), as it would be of help for some stuff I'm doing at Tor.

[evilaliv3]

sure you have simply to tell me witch file do you want to be fetched
automatically and from where and i will manage to make it available to you
already parsed as a data structure that you can use to implement your
decision alghoritm.

let me know how to proceed!

[ilv]

Hi. Despite of what I thought, I actually had to run a tor2web node and modify t2w.py in order to see if what I was doing was correct. So, this is what I have so far (tried locally):

• Detect when user add /GeTor to an url and process it.
• Detect OS and language via User-Agent and Accept-Language.
• If user is using iPhone or Android, redirect her to the corresponding product stores.
• If user is using Tor Browser, redirect it to Tor Browser's downloads page.
• If user is using Linux, show her an error message.
• If user is using either Windows or Mac OS X, send her a ZIP file, depending on her OS and language. For now, I'm assuming that the Tor Browser files had already been downloaded and are available in a folder accessible by t2w. I'm not sure if this is the intended behaviour, though.

Is this near of what you had in mind?
You can check the modifications I've made here: https://github.com/ilv/Tor2web-3.0/tree/feature/gettor

Look forward to your feedback!

[virgil]

Just make GeTor a special subdomain. So the link will be: GeTor.tor2web.org

-V

On Friday, January 30, 2015, ilv [email protected] wrote:

Hi. Despite of what I thought, I actually had to run a tor2web node and
modify t2w.py in order to see if what I was doing was correct. So, this is
what I have so far (tried locally):

• Detect when user add /GeTor to an url and process it.
• Detect OS and language via User-Agent and Accept-Language.
• If user is using iPhone or Android, redirect her to the
  corresponding product stores.
• If user is using Tor Browser, redirect it to Tor Browser's downloads
  page.
• If user is using Linux, show her an error message.
• If user is using either Windows or Mac OS X, send her a ZIP file,
  depending on her OS and language. For now, I'm assuming that the Tor
  Browser files had already been downloaded and are available in a folder
  accessible by t2w. I'm not sure if this is the intended behaviour, though.

Is this near of what you had in mind?
You can check the modifications I've made here:
https://github.com/ilv/Tor2web-3.0/tree/feature/gettor

Look forward to your feedback!

—
Reply to this email directly or view it on GitHub
#168 (comment)
.

[fpietrosanti]

@virgil with a subdomain we would not have the "plausible deniability" feature (you may be visiting your favourite porn, instead of downloading TBB). I added #202 to track your feature idea

[fpietrosanti]

@ilv Sounds fine, perfect integration!

Regarding the language, also for OS X and Windows, we shall detect the user browser version/language and provide the TBB in the proper language.

Regarding the format, i see that the Windows Format is an executable while the Mac OS X format is a DMG file and not a ZIP file.

Then we shall integrate the automatic downloader component that keep in sync the downloaded TBB in that folder with the official TBB download page.
Those could run on startup or trough an automatic Twisted scheduler:

• once every X in order to check if the local mirror of the latests version is complete.
• once every Y in order to check if there is a new TBB version, then make a new mirror, then replace the already downloaded one

We should be careful not to serve broken TBB package (ie: a user download from /GeTor when Tor2web is downloading a new package), so i suggest to first make a full-mirror of the latest-version, the made it available to the user.
Be carefu

[evilaliv3]

wait @fpietrosanti. imho all is good apart the mirroring idea.

we can simply forward the user to the relevant page that depending on the
situation can be a direct downlooad (TBB), a fucking download page with
some logic (the apple store).

downloading directly and keeping in sync the mirror of all makes tor2web
became a monster and will make it lose a lot of bandwidth (i expect TBB
servers to have a better connection than the one of a volunteer).

what do you think?

[fpietrosanti]

@evilaliv3 I think that we can only manage this feature by providing directly to the end-user the relevant TBB installation file with no redirection what-so-ever (aside from Android/Apple things).

That's a requirement because this feature is designed to work in place where torproject.org domain is blocked.

So, the latest version of the TBB installation file must be served directly from Tor2web.

In order to serve the latest version of TBB installation file, the Tor2web software must have this file already stored on it's filesystem.

In order to have it stored on the filesystem we must have it mirrored.

To have a full mirror of latest version of TBB on disk doesn't not require neither of a lot of data neither of a lot of bandwidth:

Considering the average size of TBB to be 25MB, the target platform to be Mac OS X (DMG) and Windows (EXE in 32bit and 64bit format), the amount of languages (15) we have 45 TBB installation file for about 1.1GB of disk space required/bandwidth. Considering the release of 1 new TBB per month, it's about 13.2GB of download bandwidth per year, that's nearly nothing.

[evilaliv3]

ok i see. i can manage to stream it from the upstream server (torproject.org, whatever) without problems so that we do not have to store it on the tor2web machine that can remain a lilliputian service.

[fpietrosanti]

@evilaliv3 We could serve a TBB file in streaming, but doing that way we would increase the bandwidth required when many users want to download TBB from /GeTor facility.

Not having a local mirror, to serve 10 times 1 TBB installation file in Arabic 32bit for Windows, it would require to download it 10 times from torproject.org.

Instead, having a local mirror, would require it to download from torproject.org only 1 time.

A possible solution "in the middle" is to serve the TBB installation file requested in streaming only the 1st time is requested, then keep a local cache only of that file.

By the way, i think that's much easier/quicker to have always a stored local mirror of all TBB's latest versions, because it's a more deterministic process (ie: the GeTor always expect to have a local file of latests version available in a specific path with a specific file naming) and the code written to do it can be better re-used for other purposes.

[ilv]

@fpietrosanti great, thanks! The current code actually detects the language, so no problem there. I considered a zip file because I don't know how to send two files in a row with twisted. Of course, this is considering that we would send both files, the {.exe, .dmg} and the .asc signature file (right?).

With respect to the mirroring, I also believe that the easier approach would be to have a local mirror of all Tor Browser's latest version (that's what the current code assumes). For this, a script in charge of detecting when a new version of Tor Browser is available (and download it) would be needed.

p.s. The sizes of Tor Browser for Windows and Mac OS X are about ~33Mb and ~38Mb.

[fpietrosanti]

@ilv Nice point on the .asc file!
Thinking about it, i think that we should not send a zip bundle of the TBB file + the .asc, because the purpose of this functionality is to enable a third party to create an independent (private/uncensored) download website.
So it's better to send directly to the end-user the .DMG and .EXE file, that keep usability high (having to uncompress a zip file, maybe difficult for certain set of users, such as 60yo human rights lawyer we're focusing on just now).

With the regards to the .asc file, we can think to have a simple parameters to be passed to the /GeTor in order to enable downloading the TBB file OR the signature file separately.

For example we may have URLs with parameters working as follow:

• /GetTor/file : gives you the latests TBB for your OS/Language
• /GetTor/signature : gives you the signature file for the latests TBB for your OS/Language

That way, the designer of the simplified download page, can setup a simplified download URL for both the TBB file and the signature file (in a similar way of https://www.torproject.org/projects/torbrowser.html.en).

But it's up to the user to download the signature file and verify it.

Regarding the size, assuming each TBB it's 38MB, currently the archive would require about 1.7GB of local disk storage, that's something available on a cheap $5/month VPS with 10GB of disk + 1TB/bandwidth.

However, just as a consideration for the mirroring process, i think that we shall start delivering the new TBB version, only when the batch mirroring process of the new version has been completed. That's to be 100% sure to avoid sending back to the user broken-TBB-file (ie: A user X download from Tor2web TBB file, while this file is being downloaded by Tor2web from torproject.org, so the user X receive an incomplete/broken file).

[ilv]

@fpietrosanti Ok, I see. Having a separate URL for the signature sounds fine, there should always be the option to check the integrity of Tor Browser for those who want/know how to do it.

With respect to the mirroring process, I totally agree. The script designed to do the mirroring should safely replace the old bundles after the new ones have been downloaded.

[fpietrosanti]

@ilv good! After we'll have this /GeTor feature in place we could work on #220 following the idea of Virgil & the experience done with Amnesty, trying to make a super easy/hyper-cool-UX/internationalized Tor download page that use /GeTor, including instruction for verification!

[ilv]

Changes for supporting the request of Tor Browser (GetTor/file) and its signature (GetTor/signature) can be found on: https://github.com/ilv/Tor2web-3.0/tree/feature/gettor

Since the code to process GetTor requests got larger, I took the liberty of creating a new module under utils/, minimizing the changes on t2w.py. Please note that this code considers that the files inside data/torbrowser do not have version numbers, instead they have the "latest" substring. This make things easier, but means that the script that automatically get the latest bundles should rename the files after moving the files into the download folder. What do you think of this?

[ilv]

Also, great! we should definitely work on #220 afterwards.

[fpietrosanti]

@ilv for the code review and integration @evilaliv3 is THE man!

I agree about the the use of the "latest" approach, as we're going to serve always the latests version. We may also use the approach of symlinking "latests" to the "latests version number", for example if the latests if 4.0.5, there will be a directory called 4.0.5 and a symlink "latest -> 4.0.5" .
That way when there is a new version (ie: 4.0.6), the mirroring process will start downloading TBB's in 4.0.6 and once finished, will change the "latest" symlink from 4.0.5 to 4.0.6, then delete 4.0.5 .

We could check for new version once per hour.

We shall probably do a "retry" for download of files during the mirroring before failing, in case of download error.

I think it's also useful to keep a detailed log of the mirroring and to trigger an email for un-recoverable error (ie: 3 times timeout or 3 times connection reset) ?

It's also useful to check for a minimum disk space available before starting the mirroring, considering twice the size of the full mirror.

[ilv]

Ok @fpietrosanti, that sounds good!

@evilaliv3 I'm working on a script to fetch Tor Browser here, I think it would surely help for the mirroring.

[adrelanos]

1. what kind of macical code have you found working in comparing version written that way?

https://github.com/Whonix/tb-updater/blob/master/usr/lib/tbbversion_parser

sort --unique --version-sort --reverse

do you think this alghorithm may work in general give your experience in past version names?:

Probably not forever.

first: remove the version that containt a dash

It's not guaranteed, that there will always be dashes. I should have kept a history of existing tbb version files.

second: compare the remaining ones by using from distutils.version.LooseVersion (that is deprecated and in the future will be probably replaced by Normalized Version)

Make sure it won't break if it's invalid json, any quotes or braces forgotten.

getinfo consensus/packages is my new hope to settle this once and forever. Otherwise I was considering to notify admins when RecommendedTBBVersions changed and to parse it using my brain, then manually updating a self-hosted version file. One could argue, that it would be less trustworthy. On the other hand, see also: https://trac.torproject.org/projects/tor/ticket/13065

[evilaliv3]

thank you @adrelanos: @fpietrosanti you see, it seem that the format used is not so standard :)

by the way let see what will happen with the code i've currently implemented.

[ilv]

Awesome @evilaliv3! the changes look really great, thanks :)

For 1) what about having an own tor2web.org/LatestTB url, specifying a "more standard" format, and if something happens in RecommendedTBBVersions, you make the necessary changes on tor2web.org/LatestTB to keep providing the format you specified (and thus the other nodes won't have to worry about it)?

[evilaliv3]

@ilv we won't have central nodes in the tor2web architecture; in our mind all nodes should be independent.
by the way le'ts discuss of the benefits of such a solution if you see some.

[evilaliv3]

i've resumed my experience and suggestions in implementing the gettor functionality on https://trac.torproject.org/projects/tor/ticket/14383#comment:5. this way we can try together try to help TBB specifying a usable API for RecommendedTBBVersions.

if you have any additional comments fill them on https://trac.torproject.org/projects/tor/ticket/14383

[ilv]

Well, the main benefit I see is that if/when the format of RecommendedTBBVersions change, nodes with gettor will not be affected as long as the /LatestTB adapts its result.

Of course, other option is to provide feedback and help on the ticket you mentioned. I'll see if I can add some more feedback, thanks for the link.

[fpietrosanti]

To be done the documentation at https://github.com/globaleaks/Tor2web-3.0/wiki/GetTor /cc @ilv @evilaliv3

[evilaliv3]

we should start a single unified name beteen GetTor and GeTor, i prefer the fist and all the code/wiki pages of Tor2web use currently GetTor.

@ilv take your choice ;)

[evilaliv3]

current work has been integrated by me in master and released in 3.1.39!

let's way for the bugs that will raise and then we can push out a public announcement!

[ilv]

Alright @fpietrosanti, I'll try to add my two cents in the documentation.

@evilaliv3 Yes, I noticed about the integration of GetTor in 3.1.39, great!
And I definitely prefer GetTor :)

[fpietrosanti]

This has been implemented and documented

[evilaliv3]

@fpietrosanti i've found that the API is now serving two stample release version and we are serving the oldest one; the fact that the tor recomended versions API is not well defined is a problem:

https://www.torproject.org/projects/torbrowser/RecommendedTBBVersions

[
"4.5.2",
"4.5.2-Linux",
"4.5.2-MacOS",
"4.5.2-Windows",
"4.5.3",
"4.5.3-Linux",
"4.5.3-MacOS",
"4.5.3-Windows",
"5.0a2",
"5.0a2-Linux",
"5.0a2-MacOS",
"5.0a2-Windows"
]

here currently our getttor implementation is going to use 4.5.2.

in fact we written the algorithm in order to select the oldest version expecting to have a stable and an alpha and due to the fact that we it's not easy to idcerna sable version from alpha versions.

@ilv do you have any suggestion?

[fpietrosanti]

Added ticket at Tor Project https://trac.torproject.org/projects/tor/ticket/16551

[ilv]

@fpietrosanti @evilaliv3 I totally forgot abut this, sorry. Let's keep the discussion on trac.