Saving takes a million years

[astrelsky]

Describe the bug
Ok, I'm exaggerating with this one. It doesn't take forever, it takes at most 10 seconds but it's sporadic and definitely difficult to determine what is wrong.

To Reproduce
Steps to reproduce the behavior:

1. Import large kernel
2. Run analysis.
3. Make millions of datatype and labels and data
4. Import memory dumps of the heap and device memory
5. Create uninitialized memory block, later initialize them to 0 and paste in data
6. Do some stuff and save
7. Randomly see it take forever to save?

Expected behavior
It should, believe it or not, save right away.

Screenshots
I did start recording before the save box popped up so it missed the beginning but whatever. It captured the important part I think.

example.mp4

Attachments
Not sure how helpful this will be but it's better then nothing. I'm sure everyone will be very shocked to see the AWT-EventQueue thread blocked on a database lock.

jstack.txt

Environment (please complete the following information):

• Ghidra Version: 11.5_DEV

Additional context
As with #8244 this appears to happen more often if I save immediately after making a change, which is a common thing to do if you've ever lost something for lack of saving. My laptop fans will also go brrrrrrrrrrr as well when this happens.

[dragonmacher]

The Swing thread being blocked is not itself a big deal. It looks like the background thread that is writing the program data has finished and is blocked on an OS call to sync:

   java.lang.Thread.State: RUNNABLE
	at java.io.FileDescriptor.sync0(java.base@23/Native Method)
	at java.io.FileDescriptor.sync(java.base@23/FileDescriptor.java:212)
	at db.buffers.LocalBufferFile.flush(LocalBufferFile.java:763)
	at db.buffers.LocalManagedBufferFile.setReadOnly(LocalManagedBufferFile.java:358)
	- locked <0x000000061c43ab20> (a db.buffers.LocalManagedBufferFile)
	at db.buffers.LocalManagedBufferFile.saveCompleted(LocalManagedBufferFile.java:706)

I'm curious to know if that call is taking longer than it should or if this is just where you happened to be when you grabbed the stack info.

[astrelsky]

The Swing thread being blocked is not itself a big deal. It looks like the background thread that is writing the program data has finished and is blocked on an OS call to sync:

   java.lang.Thread.State: RUNNABLE
	at java.io.FileDescriptor.sync0(java.base@23/Native Method)
	at java.io.FileDescriptor.sync(java.base@23/FileDescriptor.java:212)
	at db.buffers.LocalBufferFile.flush(LocalBufferFile.java:763)
	at db.buffers.LocalManagedBufferFile.setReadOnly(LocalManagedBufferFile.java:358)
	- locked <0x000000061c43ab20> (a db.buffers.LocalManagedBufferFile)
	at db.buffers.LocalManagedBufferFile.saveCompleted(LocalManagedBufferFile.java:706)

I'm curious to know if that call is taking longer than it should or if this is just where you happened to be when you grabbed the stack info.

It could be either or. I'll try it again tomorrow. I have a pretty good nvme drive so I don't think it would be that. I've written double digit gigabyte files in under a second or two. Then again, windows file system is notoriously crap as anyone who has ever tried to recursively iterate over huge directories on windows knows.

[astrelsky]

I wrote a little helper program to make sure I capture it. There are two saves here. The first one wasn't immediate but wasn't that long either. The second save was a nice long block. (If there are three saves, then it was the third save)

jstackdump.txt

This is the helper program I wrote like a little script kiddie

package main

import (
	"fmt"
	"io"
	"os"
	"os/exec"
	"os/signal"
	"strconv"
)

func run(path, pid string) {
	cmd := exec.Command(path, "-l", "-e", pid)
	pipe, err := cmd.StdoutPipe()
	if err != nil {
		panic(err)
	}
	err = cmd.Start()
	if err != nil {
		panic(err)
	}
	go io.Copy(os.Stdout, pipe)
	err = cmd.Wait()
	if err != nil {
		panic(err)
	}
}

func main() {
	if len(os.Args) != 2 {
		fmt.Printf("usage: %s pid\n", os.Args[0])
		return
	}

	pid := os.Args[1]

	_, err := strconv.Atoi(pid)
	if err != nil {
		fmt.Printf("usage: %s pid\n", os.Args[0])
		return
	}

	path, err := exec.LookPath("jstack")
	if err != nil {
		panic(err)
	}

	sig := make(chan os.Signal, 1)
	signal.Notify(sig, os.Interrupt)

	defer close(sig)

	for {
		select {
		case <-sig:
			return
		default:
			run(path, pid)
		}
	}

}

[astrelsky]

The Swing thread being blocked is not itself a big deal. It looks like the background thread that is writing the program data has finished and is blocked on an OS call to sync:

   java.lang.Thread.State: RUNNABLE
	at java.io.FileDescriptor.sync0(java.base@23/Native Method)
	at java.io.FileDescriptor.sync(java.base@23/FileDescriptor.java:212)
	at db.buffers.LocalBufferFile.flush(LocalBufferFile.java:763)
	at db.buffers.LocalManagedBufferFile.setReadOnly(LocalManagedBufferFile.java:358)
	- locked <0x000000061c43ab20> (a db.buffers.LocalManagedBufferFile)
	at db.buffers.LocalManagedBufferFile.saveCompleted(LocalManagedBufferFile.java:706)

I'm curious to know if that call is taking longer than it should or if this is just where you happened to be when you grabbed the stack info.

Ok yea, it might be this, It could be coming around to it a second time and waiting for the first one to finish before doing it again. Why is this necessary? There is write caching for a reason.

I'm assuming the data is very large at this point from the spike in SSD usage. Hmm yes in the project folder for this specific program I see 3 files db.1814.gbf, db.1815.gbf and tmp4805221970376045505.ps. All 3 files are 1.90 GB.

[astrelsky]

I'm assuming it is compressing the memory block data at least? I was surprised to see it was 1.9GB, unless it is using that compression algorithm which is both slow and has bad compression ratios that java seems to love for whatever reason, then I'm not really surprised at all. It's memory, so mostly repeated data (0x00, 0xff, etc) so I would expect a very high compression ratio and I only have a few GB of memory at most loaded, if even but I have to confirm.

When I first saw the tmp file, it was much smaller but when I went to confirm it's size it was 1.9 GB. So I suspect what is happening is that I save, it writes the file, starts copying it to the tmp file and then when I save again it is blocking until that copy finishes and then starts it all over again.

Thinking about it more, I think using a more modern compression scheme instead of one older than Java. I think there are zstd and snappy compressors in apache commons. Maybe one of those would be much better? (overall for ghidra, not just here)

[ghidra1]

It is know that our on-disk non-compressed storage is not space-efficient as we rely on fixed/computed block offsets into the database buffer file. This strategy greatly speeds-up random block access but greatly complicates any file compression strategy. We also employ an initial background pre-save when opening a file for update which is generally expected to complete before a save is requested with typical use. This also provides a degree of fault tolerance. This strategy assumes there will be user interaction delays when a file is first opened during which time the pre-save occurs. I hesitate to compress the entire file which would prevent random block access. We do employ compression with the Ghidra Server when such blocks are streamed between client and server.

We do not attempt to compress memory block data at the application level as this again would hinder random read/write access.

[astrelsky]

It is know that our on-disk non-compressed storage is not space-efficient as we rely on fixed/computed block offsets into the database buffer file. This strategy greatly speeds-up random block access but greatly complicates any file compression strategy. We also employ an initial background pre-save when opening a file for update which is generally expected to complete before a save is requested with typical use. This also provides a degree of fault tolerance. This strategy assumes there will be user interaction delays when a file is first opened during which time the pre-save occurs. I hesitate to compress the entire file which would prevent random block access. We do employ compression with the Ghidra Server when such blocks are streamed between client and server.

We do not attempt to compress memory block data at the application level as this again would hinder random read/write access.

Alright, then what about storing large memory blocks separately such that if they are not changed, the data is not copied?

Is there a reason everything is stored in one file? I would have expected each table to be separate. Especially for something like memory blocks where the data can be rather large, isn't changed very often and when changed, individual changes are often done together.

The only times I don't save immediately after making a change is if I know it's temporary and I want to see what happens or how something looks.

Initially I thought of compression because I thought the ".gbf" extension was ".gzf". I just exported it as a Ghidra zip file and it is 349MB, which isn't too bad considering I was just talking 💩 about that compression, assuming it is deflate.

[ghidra1]

Memory blocks are stored like everything else within the database buffer file (gbf) that is at the heart of out shared versioning and undo/redo capabilities. We do not have any provision for non-versioned or out-of-band content. Buffers within this file are versioned without domain-specific consideration which extends to our Ghidra Server interface implementation. It is very generic. Hindsight is 20/20 and if I could do it over I would certainly factor in other concepts. Believe me, I have a big list of things I would do differently in the data/project storage framework if I could start over. Working in such changes while preserving backward compatibility is a major under taking, and like all things must be prioritized against our other goals and bang-for-the-buck. Issues with the biggest-benefit and the least effort tend to get worked first. Minimal-benefit and big-effort tends to fall to the bottom.

[astrelsky]

Memory blocks are stored like everything else within the database buffer file (gbf) that is at the heart of out shared versioning and undo/redo capabilities. We do not have any provision for non-versioned or out-of-band content. Buffers within this file are versioned without domain-specific consideration which extends to our Ghidra Server interface implementation. It is very generic. Hindsight is 20/20 and if I could do it over I would certainly factor in other concepts. Believe me, I have a big list of things I would do differently in the data/project storage framework if I could start over. Working in such changes while preserving backward compatibility is a major under taking, and like all things must be prioritized against our other goals and bang-for-the-buck. Issues with the biggest-benefit and the least effort tend to get worked first. Minimal-benefit and big-effort tends to fall to the bottom.

So should I take this as a "won't fix" then? I'll probably try manually removing sections of memory that don't have any references and appear to be junk which will hopefully help.

Minimal-benefit depends on your perspective (I can't remember the word I'm looking for here). There are portions of ghidra that have been neglected for a long time and I can only assume this is why. Of course you'll see little to no benefit if you don't actually use ghidra, but the user experience matters a lot for those that do use ghidra. There are reasons people still use ida even though it costs way more than it's worth. When new versions of Ida was no longer an option, many of my teammates preferred to look into binja than try Ghidra again.

For what it's worth, I'm in complete agreement that this should be considered an edge case and may not be worth fixing right away. It is a scalability issue though.

[ghidra1]

While we added the "original file bytes" into the Program database it may be worth exploring if there is a way to store this as a non-changing static binary file. One of the biggest challenges to doing this is the Ghidra Server interface. Without our version control mechanism and server interface it would be a much easier problem to solve. But is certainly worth exploring.

[dragonmacher]

Minimal-benefit depends on your perspective (I can't remember the word I'm looking for here). There are portions of ghidra that have been neglected for a long time and I can only assume this is why. Of course you'll see little to no benefit if you don't actually use ghidra, but the user experience matters a lot for those that do use ghidra.

I'd love to see a way for users to vote on tickets. It would be nice to be able to reference our top 10/20 most in-demand change requests.