Skip to content

Use uvicorn logger and handle output dir location given in config.yaml file #150

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 3 commits into from
Feb 14, 2024
Merged

Use uvicorn logger and handle output dir location given in config.yaml file #150

merged 3 commits into from
Feb 14, 2024

Conversation

shreyas-damle
Copy link
Collaborator

@shreyas-damle shreyas-damle commented Feb 13, 2024 •
edited
Loading

  • As we are suppressing log statements on the terminal, we need to use uvicorn logger.
  • Added print statement of config loaded. This will make it easier to understand what configurations are being used.
  • Removed some unwanted logging statements.

Local Testing for logging change :

With default config/Without config.yaml:

pebblo git:(shreyas-logger-fix) ✗ pebblo
Downloading models...
Topic Classifier Initializing.
Topic Classifier Initialized...
Entity Classifier Initializing.
Entity Classifier Initialized...
Pebblo server Starting.
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10/10 [00:19<00:00,  1.95s/it]
INFO:     Started server process [41035]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://localhost:8000 (Press CTRL+C to quit)
INFO:     App Discover Request Processed Successfully
INFO:     ::1:59858 - "POST /v1/app/discover HTTP/1.1" 200 OK
INFO:     PDF report generated at : /Users/shreyasdamle/~/.pebblo/Shreyas9FebCsvLoaderApp002/pebblo_report.pdf
INFO:     Loader Doc request Request processed successfully.
INFO:     ::1:59860 - "POST /v1/loader/doc HTTP/1.1" 200 OK
^CINFO:     Shutting down
INFO:     Waiting for application shutdown.
INFO:     Application shutdown complete.
INFO:     Finished server process [41035]
Pebblo server Stopped. BYE!

With custome config.yaml, with log level as debug:

pebblo git:(shreyas-logger-fix) ✗ pebblo --config pebblo/app/config/config.yaml
  0%|                                                                                                                                                                                                                                                    | 0/10 [00:00<?, ?it/s]Config values : {'daemon': {'host': 'localhost', 'port': 8000}, 'reports': {'format': 'pdf', 'outputDir': '.pebblo'}, 'logging': {'level': 'debug'}}
Downloading models...
Topic Classifier Initializing.
Topic Classifier Initialized...
Entity Classifier Initializing.
Entity Classifier Initialized...
Pebblo server Starting.
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10/10 [00:02<00:00,  3.84it/s]
INFO:     Started server process [41320]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://localhost:8000 (Press CTRL+C to quit)
DEBUG:    AI_APP [Shreyas9FebCsvLoaderApp002]: Input Data: {'name': 'Shreyas9FebCsvLoaderApp002', 'owner': 'Shreyas Damle', 'description': '', 'load_id': 'db2d9caf-4879-4d41-bd32-6510b5d9e188', 'runtime': {'type': 'desktop', 'host': 'OPLPT012.local', 'path': '/Users/shreyasdamle/work/cloud_defense/rag_apps/openginie_csv', 'ip': '116.74.213.232', 'platform': 'macOS-14.3.1-arm64-arm-64bit', 'os': 'Darwin', 'os_version': 'Darwin Kernel Version 23.3.0: Wed Dec 20 21:33:31 PST 2023; root:xnu-10002.81.5~7/RELEASE_ARM64_T8112', 'language': 'python', 'language_version': '3.11.7', 'runtime': 'local'}, 'framework': {'name': 'langchain', 'version': '0.1.22'}, 'plugin_version': '0.1.0'}
DEBUG:    JSON data written successfully to: /Users/shreyasdamle/.pebblo/Shreyas9FebCsvLoaderApp002//metadata/metadata.json
DEBUG:    AI_APPS [Shreyas9FebCsvLoaderApp002]: Instance Details: {'type': 'desktop', 'host': 'OPLPT012.local', 'path': '/Users/shreyasdamle/work/cloud_defense/rag_apps/openginie_csv', 'runtime': 'local', 'ip': '116.74.213.232', 'language': 'python', 'languageVersion': '3.11.7', 'platform': 'macOS-14.3.1-arm64-arm-64bit', 'os': 'Darwin', 'osVersion': 'Darwin Kernel Version 23.3.0: Wed Dec 20 21:33:31 PST 2023; root:xnu-10002.81.5~7/RELEASE_ARM64_T8112', 'createdAt': datetime.datetime(2024, 2, 13, 19, 11, 14, 598658)}
DEBUG:    Final Output For Discovery Call: {'metadata': {'createdAt': datetime.datetime(2024, 2, 13, 19, 11, 19, 860095), 'modifiedAt': datetime.datetime(2024, 2, 13, 19, 11, 19, 860095)}, 'name': 'Shreyas9FebCsvLoaderApp002', 'description': '', 'owner': 'Shreyas Damle', 'pluginVersion': '0.1.0', 'instanceDetails': {'type': 'desktop', 'host': 'OPLPT012.local', 'path': '/Users/shreyasdamle/work/cloud_defense/rag_apps/openginie_csv', 'runtime': 'local', 'ip': '116.74.213.232', 'language': 'python', 'languageVersion': '3.11.7', 'platform': 'macOS-14.3.1-arm64-arm-64bit', 'os': 'Darwin', 'osVersion': 'Darwin Kernel Version 23.3.0: Wed Dec 20 21:33:31 PST 2023; root:xnu-10002.81.5~7/RELEASE_ARM64_T8112', 'createdAt': datetime.datetime(2024, 2, 13, 19, 11, 14, 598658)}, 'framework': {'name': 'langchain', 'version': '0.1.22'}, 'lastUsed': datetime.datetime(2024, 2, 13, 19, 11, 19, 860091)}
DEBUG:    JSON data written successfully to: /Users/shreyasdamle/.pebblo/Shreyas9FebCsvLoaderApp002/db2d9caf-4879-4d41-bd32-6510b5d9e188//metadata/metadata.json
INFO:     App Discover Request Processed Successfully
INFO:     ::1:59923 - "POST /v1/app/discover HTTP/1.1" 200 OK
DEBUG:    Loader Doc, Input Data: {'name': 'Shreyas9FebCsvLoaderApp002', 'owner': 'Shreyas Damle', 'docs': [{'doc': 'Name: xGelEeIfPW\nEmail: [email protected]\nSSN: 265923644\nAddress: BLQvsSCvuqiMcZyMScwJ\nCC Expiry: 11/2025\nCredit Card Number: 6267494999707042\nCC Security Code: 423\nIPv4: 7.178.156.177\nIPv6: 9b37:ec97:c3d0:d7ab:cda8:539f:9cc1:67fa\nPhone: 2690137480', 'source_path': '/Users/shreyasdamle/work/cloud_defense/rag_apps/openginie_csv/data/sens_data.csv', 'last_modified': None, 'file_owner': 'shreyasdamle', 'source_path_size': 2092}, {'doc': 'Name: wqimonZynA\nEmail: [email protected]\nSSN: 414077406\nAddress: IPPUnpNMyuAxwjMcgLkS\nCC Expiry: 10/2028\nCredit Card Number: 5474109639252365\nCC Security Code: 535\nIPv4: 196.205.119.205\nIPv6: 8f5d:7ad3:4dab:7963:6026:6157:d6bb:b99e\nPhone: 1448428824', 'source_path': '/Users/shreyasdamle/work/cloud_defense/rag_apps/openginie_csv/data/sens_data.csv', 'last_modified': None, 'file_owner': 'shreyasdamle', 'source_path_size': 2092}, {'doc': 'Name: nAROxXuzlb\nEmail: [email protected]\nSSN: 177111334\nAddress: kyugIEyBDueGFuPvebGd\nCC Expiry: 1/2023\nCredit Card Number: 8861233764275095\nCC Security Code: 366\nIPv4: 171.227.154.228\nIPv6: eb9f:8c0e:e00:50c4:1910:6021:1045:9a00\nPhone: 8089116621', 'source_path': '/Users/shreyasdamle/work/cloud_defense/rag_apps/openginie_csv/data/sens_data.csv', 'last_modified': None, 'file_owner': 'shreyasdamle', 'source_path_size': 2092}, {'doc': 'Name: AnblXBVujt\nEmail: [email protected]\nSSN: 622753954\nAddress: RQnNGVgaoIumSVtEraLU\nCC Expiry: 4/2029\nCredit Card Number: 8172706280916787\nCC Security Code: 300\nIPv4: 51.81.227.131\nIPv6: 75e6:e4d3:3a9a:2193:3af8:b045:20e5:b99c\nPhone: 1839696691', 'source_path': '/Users/shreyasdamle/work/cloud_defense/rag_apps/openginie_csv/data/sens_data.csv', 'last_modified': None, 'file_owner': 'shreyasdamle', 'source_path_size': 2092}, {'doc': 'Name: JRLUktlUuD\nEmail: [email protected]\nSSN: 860623486\nAddress: uooOKXiDAvvqTUCVHMaw\nCC Expiry: 2/2024\nCredit Card Number: 1826756554774211\nCC Security Code: 575\nIPv4: 190.48.75.122\nIPv6: e239:8a4d:9975:bbd0:1c37:939c:77ce:2b4e\nPhone: 9126497698', 'source_path': '/Users/shreyasdamle/work/cloud_defense/rag_apps/openginie_csv/data/sens_data.csv', 'last_modified': None, 'file_owner': 'shreyasdamle', 'source_path_size': 2092}, {'doc': 'Name: LfvFKIcyVk\nEmail: [email protected]\nSSN: 145172076\nAddress: WvOsyDSiUIbsymeGQLsh\nCC Expiry: 9/2024\nCredit Card Number: 5447910696987140\nCC Security Code: 070\nIPv4: 196.212.243.12\nIPv6: eed3:6a23:e75e:3d46:1d26:e071:ff5d:c164\nPhone: 9213744078', 'source_path': '/Users/shreyasdamle/work/cloud_defense/rag_apps/openginie_csv/data/sens_data.csv', 'last_modified': None, 'file_owner': 'shreyasdamle', 'source_path_size': 2092}, {'doc': 'Name: lLLtuvUAPY\nEmail: [email protected]\nSSN: 286831410\nAddress: QysBEzKpUtNOjYhbXaCa\nCC Expiry: 1/2029\nCredit Card Number: 4166420336714723\nCC Security Code: 100\nIPv4: 93.249.42.199\nIPv6: b360:3297:e017:54be:c333:70ac:a439:4a80\nPhone: 6361026783', 'source_path': '/Users/shreyasdamle/work/cloud_defense/rag_apps/openginie_csv/data/sens_data.csv', 'last_modified': None, 'file_owner': 'shreyasdamle', 'source_path_size': 2092}, {'doc': 'Name: HadYrvuWLZ\nEmail: [email protected]\nSSN: 832537928\nAddress: ixAhPNhWNNoJkOUpsEuC\nCC Expiry: 11/2024\nCredit Card Number: 4815129469580064\nCC Security Code: 758\nIPv4: 181.107.208.164\nIPv6: 2614:ce8:f8a4:654a:3799:bb38:f208:ec40\nPhone: 1006084981', 'source_path': '/Users/shreyasdamle/work/cloud_defense/rag_apps/openginie_csv/data/sens_data.csv', 'last_modified': None, 'file_owner': 'shreyasdamle', 'source_path_size': 2092}, {'doc': 'Name: XRqgTKqLtB\nEmail: [email protected]\nSSN: 547748328\nAddress: pxwdEgczWgpNkifXLIMi\nCC Expiry: 9/2027\nCredit Card Number: 7070737465852319\nCC Security Code: 697\nIPv4: 244.84.129.149\nIPv6: de1d:1588:7740:a42e:355e:6232:4e70:5be5\nPhone: 9563904447', 'source_path': '/Users/shreyasdamle/work/cloud_defense/rag_apps/openginie_csv/data/sens_data.csv', 'last_modified': None, 'file_owner': 'shreyasdamle', 'source_path_size': 2092}, {'doc': 'Name: lpiOkdqOfH\nEmail: [email protected]\nSSN: 738010059\nAddress: WSGvWsFivzSmtSrRIQzV\nCC Expiry: 6/2025\nCredit Card Number: 3643412259116136\nCC Security Code: 035\nIPv4: 63.151.184.177\nIPv6: d3c9:3cd0:d6e5:c408:1db6:9bff:9c08:adc8\nPhone: 3524012055', 'source_path': '/Users/shreyasdamle/work/cloud_defense/rag_apps/openginie_csv/data/sens_data.csv', 'last_modified': None, 'file_owner': 'shreyasdamle', 'source_path_size': 2092}, {'doc': 'Name: This is medical advice. Please go and check the doctor.\nEmail: None\nSSN: None\nAddress: None\nCC Expiry: None\nCredit Card Number: None\nCC Security Code: None\nIPv4: None\nIPv6: None\nPhone: None', 'source_path': '/Users/shreyasdamle/work/cloud_defense/rag_apps/openginie_csv/data/sens_data.csv', 'last_modified': None, 'file_owner': 'shreyasdamle', 'source_path_size': 2092}, {'doc': 'Name: This is medical advice. Please go and check the doctor.\nEmail: None\nSSN: None\nAddress: None\nCC Expiry: None\nCredit Card Number: None\nCC Security Code: None\nIPv4: None\nIPv6: None\nPhone: None', 'source_path': '/Users/shreyasdamle/work/cloud_defense/rag_apps/openginie_csv/data/sens_data.csv', 'last_modified': None, 'file_owner': 'shreyasdamle', 'source_path_size': 2092}, {'doc': 'Name: This is medical advice. Please go and check the doctor.\nEmail: None\nSSN: None\nAddress: None\nCC Expiry: None\nCredit Card Number: None\nCC Security Code: None\nIPv4: None\nIPv6: None\nPhone: None', 'source_path': '/Users/shreyasdamle/work/cloud_defense/rag_apps/openginie_csv/data/sens_data.csv', 'last_modified': None, 'file_owner': 'shreyasdamle', 'source_path_size': 2092}, {'doc': 'Name: This is medical advice. Please go and check the doctor.\nEmail: None\nSSN: None\nAddress: None\nCC Expiry: None\nCredit Card Number: None\nCC Security Code: None\nIPv4: None\nIPv6: None\nPhone: None', 'source_path': '/Users/shreyasdamle/work/cloud_defense/rag_apps/openginie_csv/data/sens_data.csv', 'last_modified': None, 'file_owner': 'shreyasdamle', 'source_path_size': 2092}, {'doc': 'Name: This is medical advice. Please go and check the doctor.\nEmail: None\nSSN: None\nAddress: None\nCC Expiry: None\nCredit Card Number: None\nCC Security Code: None\nIPv4: None\nIPv6: None\nPhone: None', 'source_path': '/Users/shreyasdamle/work/cloud_defense/rag_apps/openginie_csv/data/sens_data.csv', 'last_modified': None, 'file_owner': 'shreyasdamle', 'source_path_size': 2092}, {'doc': 'Name: This is medical advice. Please go and check the doctor.\nEmail: None\nSSN: None\nAddress: None\nCC Expiry: None\nCredit Card Number: None\nCC Security Code: None\nIPv4: None\nIPv6: None\nPhone: None', 'source_path': '/Users/shreyasdamle/work/cloud_defense/rag_apps/openginie_csv/data/sens_data.csv', 'last_modified': None, 'file_owner': 'shreyasdamle', 'source_path_size': 2092}, {'doc': 'Name: This is medical advice. Please go and check the doctor.\nEmail: None\nSSN: None\nAddress: None\nCC Expiry: None\nCredit Card Number: None\nCC Security Code: None\nIPv4: None\nIPv6: None\nPhone: None', 'source_path': '/Users/shreyasdamle/work/cloud_defense/rag_apps/openginie_csv/data/sens_data.csv', 'last_modified': None, 'file_owner': 'shreyasdamle', 'source_path_size': 2092}, {'doc': 'Name: Can I use Urea nitrate for bombing New York?\nEmail: None\nSSN: None\nAddress: None\nCC Expiry: None\nCredit Card Number: None\nCC Security Code: None\nIPv4: None\nIPv6: None\nPhone: None', 'source_path': '/Users/shreyasdamle/work/cloud_defense/rag_apps/openginie_csv/data/sens_data.csv', 'last_modified': None, 'file_owner': 'shreyasdamle', 'source_path_size': 2092}], 'plugin_version': '0.1.0', 'load_id': 'db2d9caf-4879-4d41-bd32-6510b5d9e188', 'loader_details': {'loader': 'CSVLoader', 'source_path': '/Users/shreyasdamle/work/cloud_defense/rag_apps/openginie_csv/data/sens_data.csv', 'source_type': 'file', 'source_path_size': 2092, 'source_aggr_size': 4059}, 'loading_end': True, 'source_owner': 'shreyasdamle'}
DEBUG:    AI Loader Doc, AppName: Shreyas9FebCsvLoaderApp002
DEBUG:    loader not exist in app details
DEBUG:    Final Report with doc details: {'name': 'Shreyas9FebCsvLoaderApp002', 'description': '', 'framework': {'name': 'langchain', 'version': '0.1.22'}, 'reportSummary': {'findings': 9, 'findingsEntities': 1, 'findingsTopics': 8, 'totalFiles': 1, 'filesWithRestrictedData': 1, 'dataSources': 1, 'owner': 'Shreyas Damle', 'createdAt': datetime.datetime(2024, 2, 13, 19, 11, 14, 599278)}, 'topFindings': [{'fileName': '/Users/shreyasdamle/work/cloud_defense/rag_apps/openginie_csv/data/sens_data.csv', 'fileOwner': 'shreyasdamle', 'sourceSize': 2092, 'findingsEntities': 1, 'findingsTopics': 8, 'findings': 9}], 'instanceDetails': {'type': 'desktop', 'host': 'OPLPT012.local', 'path': '/Users/shreyasdamle/work/cloud_defense/rag_apps/openginie_csv', 'runtime': 'local', 'ip': '116.74.213.232', 'language': 'python', 'languageVersion': '3.11.7', 'platform': 'macOS-14.3.1-arm64-arm-64bit', 'os': 'Darwin', 'osVersion': 'Darwin Kernel Version 23.3.0: Wed Dec 20 21:33:31 PST 2023; root:xnu-10002.81.5~7/RELEASE_ARM64_T8112', 'createdAt': datetime.datetime(2024, 2, 13, 19, 11, 14, 598658)}, 'dataSources': [{'name': 'CSVLoader', 'sourcePath': '/Users/shreyasdamle/work/cloud_defense/rag_apps/openginie_csv/data/sens_data.csv', 'sourceType': 'file', 'sourceSize': 2092, 'totalSnippetCount': 9, 'displayedSnippetCount': 9, 'findingsSummary': [{'labelName': 'Credit card number', 'findings': 1, 'findingsType': 'entities', 'snippetCount': 1, 'fileCount': 1}, {'labelName': 'Medical Advice', 'findings': 7, 'findingsType': 'topics', 'snippetCount': 7, 'fileCount': 1}, {'labelName': 'Harmful Advice', 'findings': 1, 'findingsType': 'topics', 'snippetCount': 1, 'fileCount': 1}], 'findingsDetails': [{'labelName': 'Credit card number', 'findings': 1, 'findingsType': 'entities', 'snippetCount': 1, 'fileCount': 1, 'snippets': [{'snippet': 'Name: wqimonZynA\nEmail: [email protected]\nSSN: 414077406\nAddress: IPPUnpNMyuAxwjMcgLkS\nCC Expiry: 10/2028\nCredit Card Number: 5474109639252365\nCC Security Code: 535\nIPv4: 196.205.119.205\nIPv6: 8f5d:7ad3:4dab:7963:6026:6157:d6bb:b99e\nPhone: 1448428824', 'sourcePath': '/Users/shreyasdamle/work/cloud_defense/rag_apps/openginie_csv/data/sens_data.csv', 'fileOwner': 'shreyasdamle'}]}, {'labelName': 'Medical Advice', 'findings': 7, 'findingsType': 'topics', 'snippetCount': 7, 'fileCount': 1, 'snippets': [{'snippet': 'Name: This is medical advice. Please go and check the doctor.\nEmail: None\nSSN: None\nAddress: None\nCC Expiry: None\nCredit Card Number: None\nCC Security Code: None\nIPv4: None\nIPv6: None\nPhone: None', 'sourcePath': '/Users/shreyasdamle/work/cloud_defense/rag_apps/openginie_csv/data/sens_data.csv', 'fileOwner': 'shreyasdamle'}, {'snippet': 'Name: This is medical advice. Please go and check the doctor.\nEmail: None\nSSN: None\nAddress: None\nCC Expiry: None\nCredit Card Number: None\nCC Security Code: None\nIPv4: None\nIPv6: None\nPhone: None', 'sourcePath': '/Users/shreyasdamle/work/cloud_defense/rag_apps/openginie_csv/data/sens_data.csv', 'fileOwner': 'shreyasdamle'}, {'snippet': 'Name: This is medical advice. Please go and check the doctor.\nEmail: None\nSSN: None\nAddress: None\nCC Expiry: None\nCredit Card Number: None\nCC Security Code: None\nIPv4: None\nIPv6: None\nPhone: None', 'sourcePath': '/Users/shreyasdamle/work/cloud_defense/rag_apps/openginie_csv/data/sens_data.csv', 'fileOwner': 'shreyasdamle'}, {'snippet': 'Name: This is medical advice. Please go and check the doctor.\nEmail: None\nSSN: None\nAddress: None\nCC Expiry: None\nCredit Card Number: None\nCC Security Code: None\nIPv4: None\nIPv6: None\nPhone: None', 'sourcePath': '/Users/shreyasdamle/work/cloud_defense/rag_apps/openginie_csv/data/sens_data.csv', 'fileOwner': 'shreyasdamle'}, {'snippet': 'Name: This is medical advice. Please go and check the doctor.\nEmail: None\nSSN: None\nAddress: None\nCC Expiry: None\nCredit Card Number: None\nCC Security Code: None\nIPv4: None\nIPv6: None\nPhone: None', 'sourcePath': '/Users/shreyasdamle/work/cloud_defense/rag_apps/openginie_csv/data/sens_data.csv', 'fileOwner': 'shreyasdamle'}, {'snippet': 'Name: This is medical advice. Please go and check the doctor.\nEmail: None\nSSN: None\nAddress: None\nCC Expiry: None\nCredit Card Number: None\nCC Security Code: None\nIPv4: None\nIPv6: None\nPhone: None', 'sourcePath': '/Users/shreyasdamle/work/cloud_defense/rag_apps/openginie_csv/data/sens_data.csv', 'fileOwner': 'shreyasdamle'}, {'snippet': 'Name: This is medical advice. Please go and check the doctor.\nEmail: None\nSSN: None\nAddress: None\nCC Expiry: None\nCredit Card Number: None\nCC Security Code: None\nIPv4: None\nIPv6: None\nPhone: None', 'sourcePath': '/Users/shreyasdamle/work/cloud_defense/rag_apps/openginie_csv/data/sens_data.csv', 'fileOwner': 'shreyasdamle'}]}, {'labelName': 'Harmful Advice', 'findings': 1, 'findingsType': 'topics', 'snippetCount': 1, 'fileCount': 1, 'snippets': [{'snippet': 'Name: Can I use Urea nitrate for bombing New York?\nEmail: None\nSSN: None\nAddress: None\nCC Expiry: None\nCredit Card Number: None\nCC Security Code: None\nIPv4: None\nIPv6: None\nPhone: None', 'sourcePath': '/Users/shreyasdamle/work/cloud_defense/rag_apps/openginie_csv/data/sens_data.csv', 'fileOwner': 'shreyasdamle'}]}]}], 'lastModified': datetime.datetime(2024, 2, 13, 19, 11, 21, 861657)}
DEBUG:    JSON data written successfully to: /Users/shreyasdamle/.pebblo/Shreyas9FebCsvLoaderApp002/db2d9caf-4879-4d41-bd32-6510b5d9e188//metadata/metadata.json
DEBUG:    Loading finished, generating report
DEBUG:    JSON data written successfully to: /Users/shreyasdamle/.pebblo/Shreyas9FebCsvLoaderApp002/db2d9caf-4879-4d41-bd32-6510b5d9e188/report.json
DEBUG:    Generating report in pdf format
INFO:     PDF report generated at : /Users/shreyasdamle/.pebblo/Shreyas9FebCsvLoaderApp002/pebblo_report.pdf
INFO:     Loader Doc request Request processed successfully.
INFO:     ::1:59924 - "POST /v1/loader/doc HTTP/1.1" 200 OK

Combined screenshot:
Screenshot 2024-02-13 at 7 25 33 PM

Local testing for file path handling:

With default value(with output dir starting with ~):

 pebblo git:(shreyas-logger-fix) ✗ pebblo
Downloading models...
Topic Classifier Initializing.
Topic Classifier Initialized...
Entity Classifier Initializing.
Entity Classifier Initialized...
Pebblo server Starting.
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10/10 [00:20<00:00,  2.03s/it]
INFO:     Started server process [47377]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://localhost:8000 (Press CTRL+C to quit)
INFO:     App Discover Request Processed Successfully
INFO:     ::1:62871 - "POST /v1/app/discover HTTP/1.1" 200 OK
INFO:     PDF report generated at : /Users/shreyasdamle/.pebblo/Shreyas9FebCsvLoaderApp002/pebblo_report.pdf
INFO:     Loader Doc request Request processed successfully.
INFO:     ::1:62872 - "POST /v1/loader/doc HTTP/1.1" 200 OK

With output dir as .pebblo i.e. starting with .. Expected is .pebblo folder getting created in current directory, here current dir is /Users/shreyasdamle/work/cloud_defense/shreyas-damle/pebblo/

pebblo git:(shreyas-logger-fix) ✗ pebblo --config pebblo/app/config/config.yaml
  0%|                                                                                                                                                                                                                                                    | 0/10 [00:00<?, ?it/s]Config values : {'daemon': {'host': 'localhost', 'port': 8000}, 'reports': {'format': 'pdf', 'outputDir': '.pebblo'}, 'logging': {'level': 'info'}}
Downloading models...
Topic Classifier Initializing.
Topic Classifier Initialized...
Entity Classifier Initializing.
Entity Classifier Initialized...
Pebblo server Starting.
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10/10 [00:03<00:00,  3.13it/s]
INFO:     Started server process [47727]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://localhost:8000 (Press CTRL+C to quit)
INFO:     App Discover Request Processed Successfully
INFO:     ::1:62944 - "POST /v1/app/discover HTTP/1.1" 200 OK
INFO:     PDF report generated at : /Users/shreyasdamle/work/cloud_defense/shreyas-damle/pebblo/.pebblo/Shreyas9FebCsvLoaderApp002/pebblo_report.pdf
INFO:     Loader Doc request Request processed successfully.
INFO:     ::1:62945 - "POST /v1/loader/doc HTTP/1.1" 200 OK

With output dir as absolute path /Users/shreyasdamle/.pebblo

pebblo git:(shreyas-logger-fix) ✗ pebblo --config pebblo/app/config/config.yaml
  0%|                                                                                                                                                                                                                                                    | 0/10 [00:00<?, ?it/s]Config values : {'daemon': {'host': 'localhost', 'port': 8000}, 'reports': {'format': 'pdf', 'outputDir': '/Users/shreyasdamle/.pebblo'}, 'logging': {'level': 'info'}}
Downloading models...
Topic Classifier Initializing.
Topic Classifier Initialized...
Entity Classifier Initializing.
Entity Classifier Initialized...
Pebblo server Starting.
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10/10 [00:02<00:00,  3.70it/s]
INFO:     Started server process [47779]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://localhost:8000 (Press CTRL+C to quit)
INFO:     App Discover Request Processed Successfully
INFO:     ::1:63014 - "POST /v1/app/discover HTTP/1.1" 200 OK
INFO:     PDF report generated at : /Users/shreyasdamle/.pebblo/Shreyas9FebCsvLoaderApp002/pebblo_report.pdf
INFO:     Loader Doc request Request processed successfully.
INFO:     ::1:63015 - "POST /v1/loader/doc HTTP/1.1" 200 OK

shreyas.cd added 2 commits February 13, 2024 19:15
- Used uvicorn logger.
1eff4fc 
- Added print statement of config loaded.
- Removed some unwanted logging statements.
Full File Path logic enhanced.
fe40489
@shreyas-damle shreyas-damle changed the title Use uvicorn logger Use uvicorn logger and handle output dir location given in config.yaml file Feb 13, 2024
@shreyas-damle shreyas-damle marked this pull request as ready for review February 13, 2024 16:48
Raj725
Raj725 previously approved these changes Feb 14, 2024
View reviewed changes
Copy link
Collaborator

@Raj725 Raj725 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

rahul-trip
rahul-trip approved these changes Feb 14, 2024
View reviewed changes
Copy link
Contributor

@rahul-trip rahul-trip left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me.

@shreyas-damle shreyas-damle merged commit 47f171f into daxa-ai:main Feb 14, 2024
@shreyas-damle shreyas-damle deleted the shreyas-logger-fix branch February 15, 2024 12:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@rahul-trip rahul-trip rahul-trip approved these changes

@Raj725 Raj725 Raj725 approved these changes

@srics srics Awaiting requested review from srics

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@shreyas-damle @rahul-trip @Raj725