Need Ideas. Want to sycn several ticktocks db (devices) into one influxdb 1.8 db.

[bgondell]

I've successfully implemented TickTock on Raspberry Pi Zero 2W for data collection. Looking for guidance on efficiently syncing multiple Zeros' TickTock databases to a central InfluxDB 1.8 server.

Current setup:
Data collection: 200ms intervals via Telegraf
Local TickTock flush: Every 10s
Desired sync interval: Every minute, without downsampling

Challenges:
Zeros have unreliable, metered internet connections.
Systems already under heavy load, compression might be costly.
RAM is pretty small, not many free memory available.
Considering InfluxDB output plugin approach, but unsure if optimal.

Is there a recommended strategy for periodic database synchronization while considering these bandwidth and resource constraints?

[ylin30]

Hi @bgondell, let's clarify your problem first.

if I understand you correctly, you want to set up something like this:

Local TickTock flush: Every 10s

Do u refer to TT config: tsdb.flush.frequency=10s? You don't have to if your PI-0s (and TT processes) do not die frequently. Decreasing this setting will result in more IO and CPU loads in your PI-0s.

Desired sync interval: Every minute, without downsampling

You want all the raw data to export to InfluxDB at the end.

Zeros have unreliable, metered internet connections.

How unreliable? It keeps up and down frequently, or it can keep up for a while and then down for a while?

Systems already under heavy load, compression might be costly.
RAM is pretty small, not many free memory available.

OK. I think TT should be good enough to survive the harsh condition.

Considering InfluxDB output plugin approach, but unsure if optimal.

Sorry I am not familiar with InfluxDB output plugin.

My understanding is that you want TT to be a local data cache for raw metrics data (since you can't guarantee Telegraf will be able to send them to InfluxDB due to network loss). But at every minute you still want to export the raw metrics data to InfluxDB.

It is an interesting problem. The first solution might be just not using TickTockDB at all. I know some other collectors (like TCollectors) can hold unsent data in their buffers in case of any network failure. You might want to check if Telegraf has such a feature. Since you expect your network good enough to send raw data every minute to the remote InfluxDB, Telegraf only needs a small buffer.

The second option would be to export raw data in TickTockDB to InfluxDB every minute. Since they are two different DBs, we can't migrate data in binary format. The only way is to do it semantically, i.e., to read raw data from TT, translate the data to write queries, and then apply the write queries to InfluxDB. TT provides a script ./admin/migrate_data.sh to migrate whole data in a folder (i.e., a TSDB in TT concept) to another TT (I think InfluxDB should be ok too). Since you only want data in a specific time range (from the last minute to current), you then need to manage the time range by yourself. You can read this wiki for some reference.

If you just want to migrate data between TT, it will be easy. We design TT in such a way that you just need to copy data files in a TSDB folder (by default 1 day 1 TSDB) to another file path managed by another TT. So we can easily backup data overnight.

Unfortunately TT doesn't provide data export features required by your scenarios. It makes your solution complicated.

Please let me know if you need further info or helps.

[bgondell]

Thanks for your detailed response! You've understood my use case correctly.

Regarding the flush interval - good point about IO/CPU impact. I'll test with 30/60 seconds intervals.

I should have mentioned an important detail in my original post: The PIs run local dashboards using Telegraf's WebSocket output plugin for live data visualization.

TT serves two purposes here:
Provides historical data for the local dashboard
Acts as a backup in case of power failures (preserving data beyond Telegraf's flush interval)

The main goal is to centralize data from all PIs into one powerful server. I chose InfluxDB since I already have it running.

About the connectivity:
PIs use 3G/LTE with varying reliability between units
Connections are metered, so data compression would be beneficial

Based on your response, I think the second option (exporting raw data) makes the most sense, especially since InfluxDB supports gzip compression. This should help with both the bandwidth constraints and data integrity.

Does this approach sound reasonable? Really appreciate your help and the great work on TT!

[ylin30]

Regarding the flush interval - good point about IO/CPU impact. I'll test with 30/60 seconds intervals.

Please try to use the default value (5 minutes) if possible. It is a Write Ahead Logging (WAL) setting to guarantee no data loss if TT dies unexpectedly. Basically TT writes data to memory buffers first. TT will flush memory buffers to disk if buffers are full or at flush interval if not full. There is one buffer page (256 bytes by default) per time series. One page can hold about 256 data points (about one byte or even less per data point according to our compression). I don't think you can fill up one page per 10 seconds.

[ylin30]

The main goal is to centralize data from all PIs into one powerful server. I chose InfluxDB since I already have it running.

Of coz, understand. You can always migrate to TT later if InfluxDB does not satisfies you.

About the connectivity: PIs use 3G/LTE with varying reliability between units Connections are metered, so data compression would be beneficial.

FYI TT compressed data into its binary files, about 1 byte (or even less than 1 bit in some cases unbelievably) per data point.

Based on your response, I think the second option (exporting raw data) makes the most sense, especially since InfluxDB supports gzip compression. This should help with both the bandwidth constraints and data integrity.

Yes, though there are still lots of details we need to figure out. The migrate_data.sh script actually reads TT's binary data files and translates binary data to OpenTSDB puts, and pipelines the puts to a new DB, like this:

/home/ylin30/inspect.0.12.2.arm.64bit -d /tmp/data.0.12.2/ -r | nc -q 30 localhost 6181

The first part, inspect, is something like this.

[yi-IdeaPad ticktock (nan)]$ ./bin/inspect -d ./data -r
Total number of time series: 1
Inspecting tsdb ./data/2024/11/1731888000.1731974400...
put Garage_ACP 1731923221 949 unit=W
put Garage_ACP 1731923361 971 unit=W
put Garage_ACP 1731923430 941 unit=W
put Garage_ACP 1731923289 nan unit=W
Total dps = 4
Grand Total = 4
[yi-IdeaPad ticktock (nan)]$

The second part, nc, is to do pipeline. You can install nc by youself. We also have a tool implemented in C if you don't like nc.

You may consider just scp/copying TT's data files to remote sites, and then run migrate_data.sh (basically inspect and nc). The complicated part is how to decide from which timestamps you want to migrate data, if you want to do it minute by minute. If you just want to do it once a day, it will be straightforward since TT's data files are partitioned by days. You can look at data file directory and you will know what I mean. We used to have a setting to allow users to specify the granularity of TSDB (day or hour etc) but I think we remove it.