DM-10370: Optimize server memory usage for fits images #371
Conversation
- Server memory size cut in half - Optimized drawing so that user sees working message - Optimized scrolling so the it does not block when loading images - beta value (used for asinh strech) now only computed when required - added a getBetaValue service call - Uploading fits files now show masking while uploading - removed addColorBand call from server (not used) - Added way to push out memory in cache before large data is read in - Fixed bug in computing in CsysConvert.pointInPlot with rectangular images
| @@ -59,9 +64,13 @@ | |||
| else { | |||
| Fits fits= null; | |||
| try { | |||
| if (memCache != null) { | |||
| memCache.put(key, new ObjectSizeEngineWrapper.BluffSize(fitsFile.length())); | |||
| memCache.put(key, null); | |||
There was a problem hiding this comment.
I am not quite understand what it is doing here? Why the same key was put twice with different value?
There was a problem hiding this comment.
This is how I am clear space in the cache before I actually read the file. If there is a 5G file in the cache by doing this I can potentially push out the 5G from the cache before the 5G file is read. Since the act of reading would takes 10G we would go as high as 15G. With this change it will only go as high as 10G the down to 5G.
To summarize- replace a 5G file in cache:
before:
5G => 15G => 5G
after
5G => 10G => 5G
The ObjectSizeEngineWrapper.BluffSize object appears to the cache to be big but it is not.
| sharedManager = CacheManager.create(sharedConfig.getAbsolutePath()); | ||
| try { | ||
| System.setProperty("net.sf.ehcache.sizeofengine.shared.VIS_SHARED_MEM", | ||
| "edu.caltech.ipac.firefly.server.cache.ObjectSizeEngineWrapper"); |
There was a problem hiding this comment.
I think using ObjectSizeEngineWrapper.class.getName() here would be even cleaner. It will fail on setProperty so clearing it is not necessary. It'll also help find/refactor class when needed.
DM-10370: Optimize server memory usage for fits images
This is a very difficult test it requires jconsole to monitor the memory with large fits images. It could take awhile just to get set up. Mostly the review should just look over the code.
Target code to look at:
@loitly would you look at: