Storage info sampling task error - Reason & how to handle it?

[aurorahunter]

Hi Team,

Can you provide some information about the error and what needs to be done to ensure we don't face it in production?

2018-03-12 21:53:00,003 ERROR org.smartdata.hdfs.metric.fetcher.StorageInfoSampler.run 124: Storage info sampling task error:
java.lang.NullPointerException
at org.smartdata.metastore.MetaStore.getStorageCapacity(MetaStore.java:433)
at org.smartdata.hdfs.metric.fetcher.StorageInfoSampler$InfoSamplingTask.run(StorageInfoSampler.java:111)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
(END)

[PHILO-HE]

This issue is as same as #1624 and it was solved by #1625 on trunk several days ago.

Because there are multiple threads involved in using the corresponding methods. A thread can set mapStorageCapacity as null while another thread is accessing the object through mapStorageCapacity.
Thus null pointer exception is reported in this situation.

I have applied this commit to release-1.3.2. Please update the code on your side. Thanks for your report!

[mkbhanda]

Philo-he, trying to understand, updatecache() method is invoked in nearly every method in class metastore and it has side effects on a bunch of objects, including creating a non-null mapStorageCapacity.
is it only null at start time? If only at start time, does it make sense to initialize as part of class static initialization method, and avoid all future synch operations? How else might it become null with multiple threads in the mix? These thread race condition type exceptions are scary -- what if we have not caught all such paths for all such objects?

[PHILO-HE]

I have refined the code to avoid creating object referred by mapStorageCapacity at each time when updateCache() is invoked. Some methods invoked by other threads will set mapStorageCapacity as null when the storage info is updated and flushed to DB. Only if mapStorageCapacity is null does updateCache() create an object to make the info in this object which referred by mapStorageCapacity consistent with that in DB. The new PR is #1638. Thanks!