Crash: LevelDB not created for Firestore with large data set

[urosk]

[REQUIRED] Step 2: Describe your environment

• Xcode version: 11.2.1
• Firebase SDK version: 6.13 (and older)
• Firebase Component: Firestore
• Component version: 6.13

[REQUIRED] Step 3: Describe the problem

When connecting to a specific Firestore database, the Firestore iOS SDK fails to create a local LevelDB database after a simple read from the Firestore is attempted. This causes an exception to be thrown by Firestore (app crashes) within ~1 minute, I assume when garbage collection runs the next time.

When attempting the same read from an empty Firestore (by swapping GoogleService-Info.plist), the database is correctly created.

If I disable persistence, the crash does not occur.

It looks like LevelDB intialization fails due to something specific to our Firestore. I suspect the complexity and size of our data set might be causing this issue.

Steps to reproduce:

The following exception is thrown ~1 minute after the read attempt:

FIRESTORE INTERNAL ASSERTION FAILED: Failed to iterate leveldb directory: Could not open directory ~/Library/Developer/CoreSimulator/Devices/9AC16742-2008-41BF-A6C4-DA78AA0D2EDF/data/Containers/Data/Application/645B2DFF-ACB2-4FF6-B90A-86CDD52F0DA6/Documents/firestore/__FIRAPP_DEFAULT/facilit-database-test/main (errno 2: No such file or directory) (expected iter->status().ok())

When observing the file system during the read, I can see the database file created, and instantly deleted, so I assume something goes wrong during LevelDB initialization.

It doesn't matter if the data I'm attempting to read is actually in the DB, the result is the same. On the problematic Firestore, the LevelDB files are not created, and on an empty Firestore, the LevelDB files are correctly created

We already have a working Android version of our app, which does not exhibit this issue, so it leads me to believe that this might be iOS SDK specific.

Stack trace:

> Queue : com.google.firebase.firestore (serial)
> 0	0x00007fff50b97af0 in objc_exception_throw ()
> 1	0x00007fff23c4eda8 in +[NSException raise:format:arguments:] ()
> 2	0x00007fff256c9c2a in -[NSAssertionHandler handleFailureInFunction:file:lineNumber:description:] ()
> 3	0x00000001081de6e0 in firebase::firestore::util::ObjcThrowHandler(firebase::firestore::util::ExceptionType, char const*, char const*, int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) ()
> 4	0x00000001081de2fe in firebase::firestore::util::Throw(firebase::firestore::util::ExceptionType, char const*, char const*, int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) ()
> 5	0x0000000108225ca6 in firebase::firestore::util::internal::FailAssertion(char const*, char const*, int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) ()
> 6	0x0000000108225d6d in firebase::firestore::util::internal::FailAssertion(char const*, char const*, int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, char const*) ()
> 7	0x000000010823157c in firebase::firestore::local::LevelDbPersistence::CalculateByteSize() ()
> 8	0x00000001082472c3 in firebase::firestore::local::LruGarbageCollector::Collect(std::__1::unordered_map<int, firebase::firestore::local::QueryData, std::__1::hash<int>, std::__1::equal_to<int>, std::__1::allocator<std::__1::pair<int const, firebase::firestore::local::QueryData> > > const&) ()
> 9	0x0000000108246e2b in std::__1::__function::__func<std::__1::enable_if<!(std::is_same<void, decltype(fp0())>::value), decltype(fp0())>::type firebase::firestore::local::Persistence::Run<firebase::firestore::local::LocalStore::CollectGarbage(firebase::firestore::local::LruGarbageCollector*)::$_16>(absl::lts_2019_08_08::string_view, firebase::firestore::local::LocalStore::CollectGarbage(firebase::firestore::local::LruGarbageCollector*)::$_16)::'lambda'(), std::__1::allocator<std::__1::enable_if<!(std::is_same<void, decltype(fp0())>::value), decltype(fp0())>::type firebase::firestore::local::Persistence::Run<firebase::firestore::local::LocalStore::CollectGarbage(firebase::firestore::local::LruGarbageCollector*)::$_16>(absl::lts_2019_08_08::string_view, firebase::firestore::local::LocalStore::CollectGarbage(firebase::firestore::local::LruGarbageCollector*)::$_16)::'lambda'()>, void ()>::operator()() ()
> 10	0x000000010823185b in firebase::firestore::local::LevelDbPersistence::RunInternal(absl::lts_2019_08_08::string_view, std::__1::function<void ()>) ()
> 11	0x000000010823ee65 in firebase::firestore::local::LocalStore::CollectGarbage(firebase::firestore::local::LruGarbageCollector*) ()
> 12	0x00000001082035e1 in std::__1::__function::__func<firebase::firestore::core::FirestoreClient::ScheduleLruGarbageCollection()::$_4, std::__1::allocator<firebase::firestore::core::FirestoreClient::ScheduleLruGarbageCollection()::$_4>, void ()>::operator()() ()
> 13	0x00000001081c848d in firebase::firestore::util::AsyncQueue::ExecuteBlocking(std::__1::function<void ()> const&) ()
> 14	0x00000001081deae8 in firebase::firestore::util::TimeSlot::InvokedByLibdispatch(void*) ()
> 15	0x000000010a15689c in __tsan::invoke_and_release_block(void*) ()
> 16	0x000000010a1565f2 in __tsan::dispatch_callback_wrap(void*) ()
> 17	0x000000010bd45d48 in _dispatch_client_callout ()
> 18	0x000000010bd486ba in _dispatch_continuation_pop ()
> 19	0x000000010bd5ba0f in _dispatch_source_invoke ()
> 20	0x000000010bd4c40e in _dispatch_lane_serial_drain ()
> 21	0x000000010bd4d17f in _dispatch_lane_invoke ()
> 22	0x000000010bd58a4e in _dispatch_workloop_worker_thread ()
> 23	0x00007fff51c0171b in _pthread_wqthread ()
> 24	0x00007fff51c0157b in start_wqthread ()

Relevant Code:

[schmidt-sebastian]

@urosk Thanks for your report and sorry for your troubles! Do you know if this happens on a single device, or is this happening across multiple devices? Furthermore, does the issue persist if you manually delete the LevelDB files (or invoke clearPersistence(completion:))?. Thank you!

[schmidt-sebastian]

One more follow up: Would it also be possible for you to send us the LevelDB files that are causing these problems? If it is possible, you can reach me at mrschmidt(at)google.com.

[urosk]

@schmidt-sebastian I tested it on a physical device and a couple of simulators, same issue.
The LevelDB files get auto-deleted by the SDK within 0.1 seconds of them being created. Just the root Firestore folder within iOS Documents folder remains.

Let me know if you need any further info.

[schmidt-sebastian]

This sounds scary. Is there a possibility your can share a small reproduction with us?

I also want to understand your previous statements more. You said that this might be influenced by the complexity and size of your dataset. Are you referring to the size and complexity of the size of your dataset on the backend? Or is this related to size of your data immediately before deletion?

We believe this might also be related to #4269, so we would like to prioritize a fix for this.

[urosk]

I mean that the Firestore database, which is causing issues with creating the on-device LevelDB, is populated by quite a lot of data and we have several levels of nesting of data structures. I don't know if that affects levelDB initialization in any way, but I assumed it somehow does, since the issue doesn't happen on an empty Firestore database.

I can just run the following queries from the Firestore tutorial:

let docRef = db.collection("cities").document("SF")

docRef.getDocument { (document, error) in
    if let document = document, document.exists {
        let dataDescription = document.data().map(String.init(describing:)) ?? "nil"
        print("Document data: \(dataDescription)")
    } else {
        print("Document does not exist")
    }
}

... and it fails to create the LevelDB on the populated Firestore, and succeeds on an empty Firestore (even if the data the query is requesting is not present in either).

I can try to recreate a test Firestore which causes this issue and grant you access, so you can debug this easier.

[wilhuff]

We've had no luck reproducing this. The contents of the database on the Firestore server should have no impact on LevelDB initialization.

Does this reproduce in a clean Xcode project but re-using the GoogleServices-Info.plist from your project where it's failing? If so, could you share that file privately with me? I'm [email protected].

Another alternative would be to just zip up your whole Xcode workspace and share that. You could share that temporarily via Google Drive or similar.

[urosk]

Turns out it's not a Firestore bug.

The issue was in an SQLite library which is apparently messing with the documents folder. Disabling the library fixed the issue.

Sorry for the false alarm.

[mikelehen]

@urosk Very interesting! Thanks for following up. Can you provide more info about the SQLite library? In particular I wonder if it's possible it could also be the root cause of #4269 or if there's something else going on there.

[wilhuff]

This is perhaps another argument for resolving #843.

[urosk]

I doubt this will be helpful - but The SQLite library was iterating over the documents folder and deleting all files which contained the name of the SQL database in their filename.

And by coincidence, it also pattern matched some firestore files.

So yeah, I guess the takeaway is that a lot of things (programmer errors) can go wrong in the Documents folder.