pcouy

joined 1 year ago
[–] [email protected] 1 points 1 month ago

Thank you for the link. I've seen it posted a few days ago.

The caching proxy for this tutorial should easily work with any tile server, including self-hosted. However, I'm not sure what the benefits would be if you are already self-hosting a tile server.

Lastly, the self-hosting documentation for OpenFreeMap mentions a 300GB of storage + 4GB of RAM requirement just for serving the tiles, which is still more than I can spare

[–] [email protected] 1 points 2 months ago

I can recommend some stuff I've been using myself :

  • Dolibarr as an ERP + CRM : requires some work to configure initially. As most (if not all) features are disabled by default, it requires enabling them based on what you need. It also has a marketplace with a bunch of modules you can buy
  • Gitea to manage codebases for customer projects. It can also do CI but I've not looked into it yet
  • Prometheus and its ecosystem (mostly promtail and grafana) for monitoring and alerting
  • docker mail server : makes it quite easy to self host a full mail server. The guides in their doc made it painless for me to configure dmarc/SPF/other stuff that make e-mail notoriously hard to host
  • Cal.com as a self hostable alternative to calendly
  • Authentik for single sign-on and centralized permission management
  • plausible for lightweight analytics
  • a mix of wireguard, iptables and nginx to basically achieve the same as cloudflare proxying and tunnels

I design, deploy and maintain such infrastructures for my own customers, so feel free to DM me with more details about your business if you need help with this

[–] [email protected] 7 points 2 months ago (1 children)

It's a server that hosts map data for the whole world, and sends map fragments (tiles)as pictures for the coordinates and zoom levels that clients request from them

[–] [email protected] 5 points 2 months ago

Are you talking about Nginx Plus ? It seems to be a commercial product built on top of Nginx

[–] [email protected] 5 points 2 months ago (3 children)

According to the Wikipedia article, "Nginx is free and open-source software, released under the terms of the 2-clause BSD license"

Do you have any source about it going proprietary ?

[–] [email protected] 6 points 2 months ago (5 children)

It's still available in Debian's default repositories, so it must still be open source (at least the version that's packaged for Debian)

[–] [email protected] 3 points 2 months ago

There have been some changes in a few recent releases related to the concerns I raised :

  • the default tile provider is now hosted by the Immich's team using protomaps (still uses vloudflare though)
  • a new onboarding step providing the option to disable the map feature and clarifying the implications of leaving it enabled has been added
  • the documentation has been updated to clarify how to change the map provider, and includes this guide as a community guide
 

This is a guide I wrote for Immich's documentation. It features some Immich specific parts, but should be quite easy to adapt to other use cases.

It is also possible (and not technically hard) to self-host a protomaps release, but this would require 100GB+ of disk space (which I can't spare right now). The main advantages of this guide over hosting a full tile server are :

  • it's a single nginx config file to deploy
  • it saves you some storage space since you're only hosting tiles you've previously viewed. You can also tweak the maximum cache size to your needs
  • it is easy to configure a trade-off between map freshness and privacy by tweaking the cache expiration delay

If you try to follow it, please send me some feedback on the content and the wording, so I can improve it

[–] [email protected] 0 points 2 months ago

It's not that I don't believe you, I was genuinely interested in knowing more. I don't understand what's so "precious" about a random stranger's thought on the internet if it's not backed up with any source.

Moreover, I did try searching around for this and could not find any result that seemed to answer my question.

[–] [email protected] 5 points 2 months ago (1 children)

Why do you trust NordVPN more than your ISP ? Is your ISP known to be especially bad ?

[–] [email protected] 7 points 2 months ago (6 children)

Can you give examples of countries where mainstream media is not owned by billionaires ?

[–] [email protected] 2 points 3 months ago* (last edited 3 months ago) (1 children)

I'm pretty sure they are actually hosting it. The tech is quite different (cofractal uses urls ending with {z}/{x}/{y}, while their tile sever uses this stuff that works quite differently)

[–] [email protected] 6 points 3 months ago (3 children)

They told me about hosting their own tile server earlier today. I'm really impressed by how fast they moved !

A pull request for a privacy page during the onboarding is in the works, and I've been working with them to update the settings page and documentation (with the goal of providing an easy way to switch map providers). They are also working on a privacy policy, and want to ship all of this in a few weeks as part of a single release.

Once again, I'm really impressed with how well they're handling this

 

Context : Immich default map tile provider (which gets sent a bunch of PII every time you use the map feature) is a company that I see no reason to trust. This is a follow-up to this post, with the ~~permanent~~ temporary fix I came up with. I will also summarize the general opinion from the comments, as well as some interesting piece of knowledge that commenters shared.

Hacky fix

This will use Nginx proxy module to build a caching proxy in front of Open Street Map's tileserver and to serve a custom style.json for the maps.

This works well for me, since I already proxy all my services behind a single Nginx instance. It is probably possible to achieve similar results with other reverse proxies, but this would obviously need to be adapted.

Caching proxy

Inside Nginx's http config block (usually in /etc/nginx/nginx.conf), create a cache zone (a directory that will hold cached responses from OSM) :

http {
     # You should not need to edit existing lines in the http block, only add the line below
    proxy_cache_path /var/cache/nginx/osm levels=1:2 keys_zone=osm:100m max_size=5g inactive=180d;
}

You may need to manually create the /var/cache/nginx/osm directory and set its owner to Nginx's user (typically www-data on Debian based distros).

Customize the max_size parameter to change the maximum amount of cached data you want to store on your server. The inactive parameter will cause Nginx to discard cached data that's not been accessed in this duration (180d ~ 6months).

Then, inside the server block that serves your Immich instance, create a new location block :

server {
    listen 443 ssl;
    server_name immich.your-domain.tld;

    # You should not need to change your existing config, only add the location block below

    location /map_proxy/ {
        proxy_pass https://tile.openstreetmap.org/;
        proxy_cache osm;
        proxy_cache_valid 180d;
        proxy_ignore_headers Cache-Control Expires;
        proxy_ssl_server_name on;
        proxy_ssl_name tile.openstreetmap.org;
        proxy_set_header Host tile.openstreetmap.org;
        proxy_set_header User-Agent "Nginx Caching Tile Proxy for self-hosters";
        proxy_set_header Cookie "";
        proxy_set_header Referer "";
    }
}

Reload Nginx (sudo systemctl reload nginx). Confirm this works by visiting https://immich.your-domain.tld/map_proxy/0/0/0.png, which should now return a world map PNG (the one from https://tile.openstreetmap.org/0/0/0.png )

This config ignores cache control headers from OSM and sets its own cache validity duration (proxy_cache_valid parameter). After the specified duration, the proxy will re-fetch the tiles. 6 months seem reasonable to me for the use case, and it can probably be set to a few years without it causing issues.

Besides being lighter on OSM's servers, the caching proxy will improve privacy by only requesting tiles from upstream when loaded for the first time. This config also strips cookies and referrer before forwarding the queries to OSM, as well as set a user agent for the proxy following OSM foundation's guidelines (according to these guidelines, you should add a contact information to this user agent)

This can probably be made to work on a different domain than the one serving your Immich instance, but this probably requires to add the appropriate headers for CORS.

Custom style.json

I came up with the following mapstyle :

{
  "version": 8,
  "name": "Immich Map",
  "sources": {
    "immich-map": {
      "type": "raster",
      "tileSize": 256,
      "tiles": [
        "https://immich.your-domain.tld/map_proxy/{z}/{x}/{y}.png"
      ]
    }
  },
  "sprite": "https://maputnik.github.io/osm-liberty/sprites/osm-liberty",
  "glyphs": "https://fonts.openmaptiles.org/{fontstack}/{range}.pbf",
  "layers": [
    {
      "id": "raster-tiles",
      "type": "raster",
      "source": "immich-map",
      "minzoom": 0,
      "maxzoom": 22
    }
  ],
  "id": "immich-map-dark"
}

Replace immich.your-domain.tld with your actual Immich domain, and remember the absolute path you save this at.

One last update to nginx's config

Since Immich currently does not provide a way to manually edit style.json, we need to serve it from http(s). Add one more location block below the previous one :

location /map_style.json {
    alias /srv/immich/mapstyle.json;
}

Replace the alias parameter with the location where you saved the json mapstyle. After reloading nginx, your json style will be available at https://immich.your-domain.tld/map_style.json

Configure Immich to use this

For this last part, follow steps 8, 9, 10 from this guide (use the link to map_style.json for both light and dark themes). After clearing the browser or app's cache, the map should now be loaded from your caching proxy. You can confirm this by tailing Nginx's logs while you zoom and move around the map in Immich

Summary of comments from previous post

Self-hosting a tile server is not realistic in most cases

People who have previously worked with maps seem to confirm that there are no tile server solution lightweight enough to be self hosted by hobbyists. There is maybe some hope with generating tiles on demand, but someone with deep knowledge of the file formats involved in the process should confirm this.

Some interesting links were shared, which seem to confirm this is not realistically self-hostable with the available software :

General sentiment about this issue

In all this part, I want to emphasize that while there seems to be a consensus, this is only based on the few comments from the previous post and may be biased by the fact that we're discussing it on a non-mainstream platform. If you disagree with anything below, please comment this post and explain your point of view.

  • Nobody declared that they had noticed the requests to a third-party server before
  • A non-negligible fraction of Immich users are interested in the privacy benefits over other solutions such as Google photos. These users do not like their self-hosted services to send requests to third-party servers without warning them first
  • The fix should consist of the following :
    • Clearly document the implications of enabling the map, and any feature that sends requests to third parties
    • Disable by default features that send requests to third parties (especially if it contains any form of geolocated data)
    • Provide a way to easily change the tile provider. A select menu with a few pre-configured style.json would be nice, along with a way to manually edit style.json (or at least some of its fields) directly from the Immich config page
272
submitted 3 months ago* (last edited 3 months ago) by [email protected] to c/[email protected]
 

Update : I made a follow-up post containing a Nginx-based solution to cache map tiles from OSM and limit the amount of PII you send

While monitoring the logs in Rethink DNS (awesome app BTW) today, I noticed the Immich app making requests to api-l.cofractal.com.

After reaching out on Immich's discord, the devs explained to me that it is used as a tile provider for the map feature. I can confirm it is not realistic to self-host a tile provider without heavily tuning down the level of details on the map (which would still require a lot of disk space and CPU time). I understand the need for a third-party service to provide the map tiles, but I'm concerned by this one.

Visiting cofractal.com only tells us that they're selling APIs. I did not find any details about the company, not even the country they're registered in. The website is also missing informations about what they are logging or not. Everything else seems gated behind a login page, but they "are not currently accepting new customers". The whois for the domain says they're in California. Digging a bit more, I find AS26073 which apparently is the same company.

This bothers me, because Cofractal gets sent every location you viewed (and the zoom level) on Immich's map, along with your client's IP address and a "Referrer" header pointing to your Immich instance. This sounds like a lot of PII to me. It's also behind cloudflare which gets to see the same stuff.

When asked about it, one dev (thanks to them for almost instantly replying to every concern/question I threw at them) explained that they personally know the people behind Cofractal. According to this Immich dev, Cofractal provides free access to its paid service to Immich's user base as a way to support the project, with the side benefit of load testing their platform.

This explanations seems plausible and reasonable to me. However, I do not personally know the people behind Cofractal, and by default, I do not trust for-profit companies to act in an altruistic way. Here's a summary of everything that makes me uneasy about this company :

  • it does not say anything about the kind of data they are logging or not
  • it requires digging through whois records to find the most basic info about the company
  • it freely provides access to its normally paid service (for the whole Immich user base), but it does not communicate about it (or it is really hard to find)
  • it does not communicate about anything : searching for its name only returns its home page and websites with informations on Autonomous Systems
  • it is "not currently accepting new [paying] customers" while providing the service for free to a quite large user base (Immich v1.109.2 got 170k downloads in 5 days, v1.108.0 got 438k downloads in 13 days )
  • It is not mentioned anywhere in the whole immich.app website (searching for site:immich.app "cofractal" gave me no result). Not even a "Thank You" or "Sponsor" note on the homepage for the free API
  • (it is behind cloudflare)

The dev I talked to encouraged me to create a feature request, and seemed favorable to adding a switch for disabling maps client side. It is already possible to disable it server-wide, and the "URL to a style.json map theme" option seems to provide a way to customize the tile provider. Which leads to this post : I'm trying to collect feedback on this before creating the feature request.

  • It should be made prominently clear to server admins that leaving maps enabled causes clients to send requests to a third party-server and give details about what is sent (viewed locations, zoom level, IP address, Immich instance URL). The Post Install Steps in the docs and a paragraph above the switch on the config page seem like good places to me. Are there other/more appropriate place for such a warning ?
  • The "URL to a style.json map theme" option should probably be renamed to make it clearer that it allows changing tile providers. Or better yet, it could be reworked to make it easier to choose which third-party you decide to trust
  • What do you think about the idea of providing instance admins with a list of choices for tile providers ? Maybe with a short pros/cons list in the docs for each provider. I'd be fine with using a more reputable provider with the extra step of configuring my own API key (which would probably require proxying requests to the tile provider to not share the API key with all clients)
  • Should the Immich server proxy requests to the tile provider in any case ? Since the tile provider has access to the Referrer and Origin headers (which is probably required for CORS), they are currently able to link user IP addresses with Immich instances. Proxying requests with the Immich server should prevent that.
  • I would go as far as making maps disabled by default for new installs. I understand that "disabling by default would be a significant downgrade for a majority of users", but I feel like there's a strong overlap between the self-hosting and privacy communities. So we should at least have some debate about it

I've also been told that I'm the first one to raise concerns about this, which leads to one more question : Did nobody complain because nobody noticed ? Or are my concerns unjustified ?

view more: next ›