OpenSearch Integration

[macohen]

Hi,

Wondering if you've considered integration with OpenSearch (https://www.opensearch.org). It's probably pretty similar to the ElasticSearch integration given that we're an Apache 2.0 fork of ElasticSearch. Also, if this might interest you, we're looking to make integration with OpenSearch specifically for search use cases easier and have an RFC out for a pipeline that takes in a search query, allows a developer to apply transformations to the query, retrieves results, and then allows the developer to apply transformations to results before returning to the user. Search Pipelines: opensearch-project/search-processor#80. Take a look, feel free to leave feedback.

Thanks,
Mark

[sidprobstein]

Hi @macohen absolutely! I suspect the RequestsGet connector can be configured, but will def prefer the official clients...

Thanks very much also for the pointer... we will review it and comment.

SWIRL 1.8 just added processing pipelines, and some processors for dupe detection 😄
https://github.com/sidprobstein/swirl-search/wiki/3.-Developer-Guide#processing-pipelines

[sidprobstein]

Just posted a working OpenSearch connector. There's a SearchProvider for OpenSearch with the Enron Email also.
https://github.com/sidprobstein/swirl-search/wiki/3.-Developer-Guide#enron-email-data-set

Result example:

{
            "swirl_rank": 5,
            "swirl_score": 104.74130467848791,
            "searchprovider": "ENRON Email (local/OpenSearch)",
            "searchprovider_rank": 1,
            "title": "Forbes Article-Gas Fired Power Plants",
            "url": "allen-p/deleted_items/124.",
            "body": "Don't know if you saw this already. Interesting.KenIndigestion Daniel Fisher, Forbes Magazine, 10.29.01 All those new natural gas-fired electric plants mean more gas demand, right? Maybe not.For natural gas bulls, there was never a surer sign of good times to come than the order book at General Electric Corp. Backlogs in GE's Power Systems group almost tripled to $25 billion in the late 1990s as scores of utilities and freestanding power producers ordered natural gas turbines to supply the nation's growing appetite for electricity. Gas demand had to go up as all those shiny new turbines were connected to the grid. Ah, but there was a flaw in that argument--one that is becoming glaringly clear as the price of gas plunges below $2 per million Btu at the wellhead from as high as $10 last year. (A million Btu of gas equals just about 1,000 cubic feet.) Those new gas plants were supposed to steal business from pollution-spewing coal units, which still supply half the nation's electricity. But in a painstaking study of 788 power plants that supply the bulk of U.S. electricity, Charles Studness, president of Studness Research, found quite the opposite could happen. Studness determined that 79% of the \"new\" electricity supplied to the grid over the past five years came from existing plants, mostly coal-fired units. And there's plenty more where that came from. Studness figures existing coal plants have the potential to increase output by 50 million megawatt-hours a year for the next five years, representing about half the expected 2.5%-a-year increase in demand. Coal's advantage over gas: It's cheap. Utilities paid an average of $1.20 per million Btu for coal last year, compared with $4.30 for gas (delivered). Even with depressed gas prices, many coal plants are still cheaper to operate. So while those new turbines being installed by Calpine (nyse: CPN - news - people) and other independent power producers, (see table, below) will burn a lot of gas, they will do it mostly at the expense of older gas plants that burn half again as much. Addressing a Burning Issue If natural gas prices stay down or merely remain volatile it will help companies that burn, trade or store gas and hurt those that sell it or provide equipment to the industry. Overleveraged gas producers and service companies could have trouble making it through the downturn. WINNERS Sales ($bil) Remarks Calpine $4.6 Its gas power plants won't compete with coal for years Chesapeake *Energy* 0.9 Hedged gas production through December 2003 Dynegy 43.4 Owns gas, coal power plants; stores, trades gas Peabody *Energy* 2.7 World's largest coal company LOSERS Sales ($bil) Remarks Devon *Energy* 3.3 Gas-heavy; $8 billion in ...debt General Electric 129.5 Sells gas turbines Pioneer Natural Resources 1.0 $1.5 billion in ...debt; 53% of production is gas Transocean SedcoForex 1.9 $4 billion in debt; heavy gas-drilling exposure Latest 12 months. Source: Market Guide via FactSet Research Systems. \"It's not our game to compete with coal or nuclear,\" says Ron A. Walter, senior vice president of San Jose, Calif.-based Calpine, which plans over the next few years to build gas-fired plants totaling 31,500 megawatts. \"It's our game to displace older gas.\" Combined-cycle gas turbines, which use hot exhaust gases from one turbine to generate steam to turn a second one, can transform 6,800 Btu of gas into a kilowatt-hour of electricity, enough to light one lightbulb for ten hours. The oldest gas plants burn as much as 13,000 Btu, and the national average is 10,500. Substitute enough new plants for old, and U.S. gas consumption, currently 62 billion cubic feet a day, could drop by as much as 1 billion cubic feet a day. That scenario sounds unlikely to Stephen Bergstrom, president of Houston-based Dynegy, an energy-trading firm that also owns both coal- and gas-fired power plants. Dynegy's 35-year-old coal plants are \"huge capital hogs,\" Bergstrom says, that *cost* ten times as much to maintain as modern gas units. Throw in the *cost* of scrubbers and other required pollution controls, and many smaller coal plants will have to be shut down, Bergstrom says. But coal-plant owners can be ingenious. Witness Midwest Generation, a nonregulated subsidiary of Edison International that bought six coal plants from troubled Commonwealth Edison of Chicago in 1999. By changing operating procedures and burning a higher grade of coal, Midwest boosted output from the 40-year-old plants 12% last year--while reducing emissions. John Long, vice president and chief technical officer of the Midwest Generation unit, thinks he can coax another 15% increase out of those aging boilers over the next five years without running afoul of environmental rules. Long term, the bullish scenario still holds: Gas will displace coal. In the meantime, gas producers have a lot of bonds to retire.",
            "date_published": "2001-10-19 13:08:22",
            "date_retrieved": "2023-01-18 22:29:43.573717",
            "author": "...",
            "payload": {
                "_source.to": "...redacted..."
            },
            "explain": {
                "stems": "energi cost",
                "body": {
                    "energy_cost_*": 0.5193583338623504,
                    "Energy_408": 0.5602530850086551,
                    "Energy_427": 0.6881646648695164,
                    "Energy_438": 0.7891793409277676,
                    "cost_644": 0.6199846012371107,
                    "result_length_adjust": 0.03485424588086185,
                    "query_length_adjust": 1.0
                }
            }
        }

This will be rolled up in 1.8.5.

[sidprobstein]

Released in 1.8.2