Skip to content

Commit b8dbf53

Browse files
fritzgrabofritzgrabo
committed
Support Elisp forms in :input header argument
1 parent 00d4388 commit b8dbf53

File tree

5 files changed

+328
-17
lines changed

5 files changed

+328
-17
lines changed

‎README.org

+4-3
Original file line numberDiff line numberDiff line change
@@ -48,9 +48,10 @@ as specifying the result's type and formatting, the various options for
4848
passing variables into the query, querying results of an Org source
4949
block, and more!
5050

51-
- ~:input~ -- the list of data sources to query. A data source can be a
52-
filename or the name of an Org reference that builds tabular or list
53-
data (an Org table, an Org source block, etc.).
51+
- ~:input~ -- the list of data sources to query. A data source can be [[examples/README.org#getting-started][a
52+
filename]], the [[examples/README.org#querying-org-references-in-local-or-other-files][name of an Org reference]] that builds tabular or list
53+
data (an Org table, an Org source block, etc.), or [[examples/README.org#querying-results-of-elisp-forms][an Elisp form]] that
54+
generates any of the above or the data to query directly.
5455

5556
- ~:cache~ -- set to ~yes~ to enable ~dsq~'s [[https://github.com/multiprocessio/dsq#caching][caching feature]].
5657

‎examples/README-unescaped.org

+120
Original file line numberDiff line numberDiff line change
@@ -120,6 +120,126 @@ LIMIT 5
120120
| [email protected] | dicta deserunt tempore |
121121
| [email protected] | doloribus quibusdam molestiae a |
122122

123+
** Querying results of Elisp forms
124+
125+
This is where it get's a little meta: it's possible to pass Elisp forms
126+
to the ~:input~ header argument, as long as they evaluate to a either a
127+
single value that is a valid ~:input~ header argument, or a list of
128+
values, each of which is either a valid ~:input~ header argument or
129+
tabular data (that is, a list of lists).
130+
131+
Let's unpack this step by step.
132+
133+
*** Single data source
134+
135+
Here's an example that queries the ~colors~ Org table from above:
136+
137+
#+begin_src dsq :input (concat "col" "ors")
138+
SELECT name FROM {}
139+
#+end_src
140+
141+
#+RESULTS:
142+
| name |
143+
|-------|
144+
| Blue |
145+
| Red |
146+
| Green |
147+
148+
*** List of data sources
149+
150+
Here's an Elisp form that evaluates to a list of data sources to query:
151+
152+
#+begin_src dsq :input `("people.json" ,(concat "col" "ors"))
153+
SELECT people.name AS name, colors.name AS color
154+
FROM {0} people
155+
INNER JOIN {1} colors ON people.id = colors.person_id
156+
#+end_src
157+
158+
#+RESULTS:
159+
| name | color |
160+
|-------+-------|
161+
| Alice | Blue |
162+
| Bob | Red |
163+
| Bob | Green |
164+
165+
*** Mixed list of data sources and tabular data
166+
167+
It's also possible to either define tabular data to query or to call
168+
functions that generate such data on the fly. Consider this a shortcut
169+
to referencing an Org source block that defines or generates data.
170+
171+
Note that for this to work, the tabular data needs to be an element of a
172+
wrapping list; it can't be passed in as a ~:input~ header argument
173+
directly, because the individual "rows" would be considered one data
174+
source each, like in the examples above.
175+
176+
Here's what that would look like for tabular data defined inline:
177+
178+
#+begin_src dsq :input '("people.json" (("person_id" "name") (1 "Blue") (2 "Red") (2 "Green")))
179+
SELECT people.name AS name, colors.name AS color
180+
FROM {0} people
181+
INNER JOIN {1} colors ON people.id = colors.person_id
182+
#+end_src
183+
184+
#+RESULTS:
185+
| name | color |
186+
|-------+-------|
187+
| Alice | Blue |
188+
| Bob | Red |
189+
| Bob | Green |
190+
191+
*** Dynamically generated tabular data
192+
193+
And finally, let's do an example that calls a function to generate the
194+
data to query on the fly.
195+
196+
Assume you have defined a simple ~org-extract~ function which uses the
197+
fabulous [[https://github.com/alphapapa/org-ql][org-ql package]] to [[https://github.com/alphapapa/org-ql#function-org-ql-select][fetch headlines from Org files]] for an org-ql
198+
query and continues to extract their meta-data and custom properties as
199+
tabular data:
200+
201+
#+begin_src elisp
202+
(defun org-extract (files &optional query)
203+
"Extract meta-data and custom properties for headings in FILES matching QUERY."
204+
(let ((headlines (org-ql-select files query))
205+
keywords)
206+
;; collect unique property keywords
207+
(mapcar (lambda (headline)
208+
(cl-loop for (keyword . _value) on (cadr headline) by #'cddr
209+
unless (member keyword keywords)
210+
do (push keyword keywords)))
211+
headlines)
212+
(cons
213+
;; header row: normalized column names
214+
(mapcar (lambda (keyword)
215+
(substring (downcase (symbol-name keyword)) 1))
216+
keywords)
217+
;; data rows
218+
(mapcar (lambda (headline)
219+
(mapcar (lambda (keyword)
220+
(let ((value (plist-get (cadr headline) keyword)))
221+
(if (or (stringp value) (numberp value))
222+
value
223+
(format "%s" value))))
224+
keywords))
225+
headlines))))
226+
#+end_src
227+
228+
Let's sum up story points of tickets that are still "ready" to be worked
229+
on in this week's ~sprint.org~ ([[https://raw.githubusercontent.com/fritzgrabo/ob-dsq/main/examples/sprint.org][raw view]]) by assignee and component to
230+
find out if we'd better reassess the ticket distribution among the team:
231+
232+
#+begin_src dsq :input `(,(org-extract "sprint.org" '(todo "READY")))
233+
SELECT assignee, component, SUM("story-points") AS points FROM {} GROUP BY assignee, component
234+
#+end_src
235+
236+
#+RESULTS:
237+
| assignee | component | points |
238+
|----------+-----------+--------|
239+
| Fritz | Backend | 5 |
240+
| Fritz | Frontend | 2 |
241+
| Rainer | Frontend | 1 |
242+
123243
** Querying JSON data with irregular attributes
124244

125245
Queried objects in JSON data might contain "irregular" attributes. For

‎examples/README.org

+129
Original file line numberDiff line numberDiff line change
@@ -132,6 +132,135 @@ LIMIT 5
132132
| [email protected] | doloribus quibusdam molestiae a |
133133
#+end_src
134134

135+
** Querying results of Elisp forms
136+
137+
This is where it get's a little meta: it's possible to pass Elisp forms
138+
to the ~:input~ header argument, as long as they evaluate to a either a
139+
single value that is a valid ~:input~ header argument, or a list of
140+
values, each of which is either a valid ~:input~ header argument or
141+
tabular data (that is, a list of lists).
142+
143+
Let's unpack this step by step.
144+
145+
*** Single data source
146+
147+
Here's an example that queries the ~colors~ Org table from above:
148+
149+
#+begin_src org
150+
,#+begin_src dsq :input (concat "col" "ors")
151+
SELECT name FROM {}
152+
,#+end_src
153+
154+
,#+RESULTS:
155+
| name |
156+
|-------|
157+
| Blue |
158+
| Red |
159+
| Green |
160+
#+end_src
161+
162+
*** List of data sources
163+
164+
Here's an Elisp form that evaluates to a list of data sources to query:
165+
166+
#+begin_src org
167+
,#+begin_src dsq :input `("people.json" ,(concat "col" "ors"))
168+
SELECT people.name AS name, colors.name AS color
169+
FROM {0} people
170+
INNER JOIN {1} colors ON people.id = colors.person_id
171+
,#+end_src
172+
173+
,#+RESULTS:
174+
| name | color |
175+
|-------+-------|
176+
| Alice | Blue |
177+
| Bob | Red |
178+
| Bob | Green |
179+
#+end_src
180+
181+
*** Mixed list of data sources and tabular data
182+
183+
It's also possible to either define tabular data to query or to call
184+
functions that generate such data on the fly. Consider this a shortcut
185+
to referencing an Org source block that defines or generates data.
186+
187+
Note that for this to work, the tabular data needs to be an element of a
188+
wrapping list; it can't be passed in as a ~:input~ header argument
189+
directly, because the individual "rows" would be considered one data
190+
source each, like in the examples above.
191+
192+
193+
Here's what that would look like for tabular data defined inline:
194+
195+
#+begin_src org
196+
,#+begin_src dsq :input '("people.json" (("person_id" "name") (1 "Blue") (2 "Red") (2 "Green")))
197+
SELECT people.name AS name, colors.name AS color
198+
FROM {0} people
199+
INNER JOIN {1} colors ON people.id = colors.person_id
200+
,#+end_src
201+
202+
,#+RESULTS:
203+
| name | color |
204+
|-------+-------|
205+
| Alice | Blue |
206+
| Bob | Red |
207+
| Bob | Green |
208+
#+end_src
209+
210+
*** Dynamically generated tabular data
211+
212+
And finally, let's do an example that calls a function to generate the
213+
data to query on the fly.
214+
215+
Assume you have defined a simple ~org-extract~ function which uses the
216+
fabulous [[https://github.com/alphapapa/org-ql][org-ql package]] to [[https://github.com/alphapapa/org-ql#function-org-ql-select][fetch headlines from Org files]] for an org-ql
217+
query and continues to extract their meta-data and custom properties as
218+
tabular data:
219+
220+
#+begin_src elisp
221+
(defun org-extract (files &optional query)
222+
"Extract meta-data and custom properties for headings in FILES matching QUERY."
223+
(let ((headlines (org-ql-select files query))
224+
keywords)
225+
;; collect unique property keywords
226+
(mapcar (lambda (headline)
227+
(cl-loop for (keyword . _value) on (cadr headline) by #'cddr
228+
unless (member keyword keywords)
229+
do (push keyword keywords)))
230+
headlines)
231+
(cons
232+
;; header row: normalized column names
233+
(mapcar (lambda (keyword)
234+
(substring (downcase (symbol-name keyword)) 1))
235+
keywords)
236+
;; data rows
237+
(mapcar (lambda (headline)
238+
(mapcar (lambda (keyword)
239+
(let ((value (plist-get (cadr headline) keyword)))
240+
(if (or (stringp value) (numberp value))
241+
value
242+
(format "%s" value))))
243+
keywords))
244+
headlines))))
245+
#+end_src
246+
247+
Let's sum up story points of tickets that are still "ready" to be worked
248+
on in this week's ~sprint.org~ ([[https://raw.githubusercontent.com/fritzgrabo/ob-dsq/main/examples/sprint.org][raw view]]) by assignee and component to
249+
find out if we'd better reassess the ticket distribution among the team:
250+
251+
#+begin_src org
252+
,#+begin_src dsq :input `(,(org-extract "sprint.org" '(todo "READY")))
253+
SELECT assignee, component, SUM("story-points") AS points FROM {} GROUP BY assignee, component
254+
,#+end_src
255+
256+
,#+RESULTS:
257+
| assignee | component | points |
258+
|----------+-----------+--------|
259+
| Fritz | Backend | 5 |
260+
| Fritz | Frontend | 2 |
261+
| Rainer | Frontend | 1 |
262+
#+end_src
263+
135264
** Querying JSON data with irregular attributes
136265

137266
Queried objects in JSON data might contain "irregular" attributes. For

‎examples/sprint.org

+55
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,55 @@
1+
# -*- bug-reference-bug-regexp: "\\b\\(\\(EX-[0-9]+\\)\\)"; bug-reference-url-format: "https://example.com/tickets/%s"; -*-
2+
3+
#+startup: showeverything
4+
#+todo: READY PROGRESS REVIEW QA PAUSED | DONE CANCELLED
5+
6+
* Tickets
7+
** READY [#C] EX-1234 Fix typo in logout message
8+
:PROPERTIES:
9+
:Assignee: Rainer
10+
:Story-Points: 1
11+
:Component: Frontend
12+
:END:
13+
14+
The logout screen says "locked out" instead of "logged out".
15+
16+
** DONE [#A] EX-1255 Add index on ~upc~ column to ~products~ table
17+
:PROPERTIES:
18+
:Assignee: Rainer
19+
:Story-Points: 2
20+
:Component: Backend
21+
:END:
22+
23+
Querying products by UPC is really slow.
24+
Let's add an index on that column.
25+
26+
** REVIEW [#A] EX-1212 Refactor authentication logic for readability
27+
:PROPERTIES:
28+
:Assignee: Fritz
29+
:Story-Points: 2
30+
:Component: Backend
31+
:END:
32+
33+
Authentication code was done in a hurry.
34+
Let's take some time to clean this up.
35+
36+
** READY [#B] EX-1132 CSV download of queried products
37+
:PROPERTIES:
38+
:Assignee: Fritz
39+
:Story-Points: 2
40+
:Component: Frontend
41+
:END:
42+
43+
On the product query result view, allow for downloading the unpaginated list of results as a CSV file.
44+
See EX-1133 for the Backend part of this.
45+
46+
** READY [#B] EX-1133 Add support for CSV format in product list API endpoint
47+
:PROPERTIES:
48+
:Assignee: Fritz
49+
:Story-Points: 5
50+
:Component: Backend
51+
:END:
52+
53+
Use the ~Content-Disposition: attachment; filename="..."~ HTTP header to suggest a file name.
54+
The file name should be the current date and time in UTC in an ISO8601-compatible format.
55+
See EX-1132 for how this is going to be called from the Frontend.

‎ob-dsq.el

+20-14
Original file line numberDiff line numberDiff line change
@@ -251,34 +251,40 @@ for expansion of the body.")
251251
(let* ((input-param (if (symbolp input-param) (symbol-name input-param) input-param))
252252
(reference (split-string input-param org-babel-dsq-format-separator)))
253253
(cons (org-babel-dsq--reference-to-temp-file (car reference) (cadr reference)) (list 'temp-file))))
254+
((listp input-param)
255+
(cons (org-babel-dsq--write-temp-file (orgtbl-to-csv input-param nil) "csv") (list 'temp-file)))
254256
(t (error "Don't know how to handle input %s: file or reference expected" input-param))))
255257

256258
(defun org-babel-dsq--reference-to-temp-file (reference fmt)
257259
"Resolve Org REFERENCE and write it to a temporary FMT file."
258-
(let ((content (org-babel-ref-resolve reference)))
259-
(unless content
260-
(error "Resolving input reference %s yielded no content" reference))
260+
(let ((data (org-babel-ref-resolve reference)))
261+
(unless data
262+
(error "Resolving input reference %s yielded no data" reference))
261263

262-
(when (listp content)
264+
(when (listp data)
263265
(if (not (or (null fmt) (string= fmt "csv")))
264266
(error "Tabular/list data in input reference %s requires csv format, but %s requested" reference fmt)
265267
(setq fmt "csv")
266-
(setq content (orgtbl-to-csv content nil))))
268+
(setq data (orgtbl-to-csv data nil))))
267269

268-
(when (and (null fmt) (stringp content))
269-
(setq fmt (org-babel-dsq--detect-format-from-content-fragment
270-
(substring content 0 (min 1000 (length content))))))
270+
(when (and (null fmt) (stringp data))
271+
(setq fmt (org-babel-dsq--detect-format-from-data-fragment
272+
(substring data 0 (min 1000 (length data))))))
271273

272274
(when (null fmt)
273275
(error "Cannot defer format for input reference %s; use '%s:<format>'" reference reference))
274276

275-
(let ((temp-file (org-babel-temp-file "dsq-" (concat "." fmt))))
276-
(with-temp-file temp-file
277-
(insert content))
278-
temp-file)))
277+
(org-babel-dsq--write-temp-file data fmt)))
279278

280-
(defun org-babel-dsq--detect-format-from-content-fragment (fragment)
281-
"Detect format of content FRAGMENT."
279+
(defun org-babel-dsq--write-temp-file (data fmt)
280+
"Write DATA to a temporary FMT file."
281+
(let ((temp-file (org-babel-temp-file "dsq-" (concat "." fmt))))
282+
(with-temp-file temp-file
283+
(insert data))
284+
temp-file))
285+
286+
(defun org-babel-dsq--detect-format-from-data-fragment (fragment)
287+
"Detect format of data FRAGMENT."
282288
(cond
283289
((string-match "\\`\\(^[[:space:]]*\\(#.*\\)?\n\\)*[[:space:]]*[{\\[]" fragment) "json")
284290
((string-match "\\`\\(^[[:space:]]*\n\\)*[^\n]*," fragment) "csv")))

0 commit comments

Comments
 (0)