Skip to content

Add caching based on file content and external references #752

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 42 commits into from
Feb 12, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
42 commits
Select commit Hold shift + click to select a range
b9f7335
wip #370
ajafff May 2, 2019
743da3e
Merge branch 'master' into cache
ajafff Jan 22, 2021
2b4a704
wip
ajafff Jan 25, 2021
6609d2b
cleanup
ajafff Jan 27, 2021
1ad610d
handle absolute paths in compilerOptions
ajafff Jan 27, 2021
780bea5
major rewrite and fixing
ajafff Jan 28, 2021
070dc38
minor adjustments
ajafff Jan 28, 2021
318f24c
make it build again
ajafff Jan 30, 2021
1b73a0a
remove dependency on ProjectHost
ajafff Jan 30, 2021
ef476f9
handle useSourceOfProjectReferenceRedirect
ajafff Jan 30, 2021
15b090a
small refactoring
ajafff Jan 31, 2021
2bb6290
cannot use --cache without --project
ajafff Feb 6, 2021
d3ad02b
revert unnecessary changes to file-filter
ajafff Feb 6, 2021
9463d2d
Merge branch 'master' into cache
ajafff Feb 6, 2021
00ce2b5
Add tests for StatePersistence
ajafff Feb 6, 2021
42e5a63
add test for argparse
ajafff Feb 6, 2021
f2fbb4b
move config hashing to runner
ajafff Feb 6, 2021
bce5c9a
don't include empty dependenies object in cache
ajafff Feb 6, 2021
97dd546
correctly mark files without dependencies as up-to-date
ajafff Feb 6, 2021
c2eafaf
use emptyArray constant where possible
ajafff Feb 6, 2021
f460075
fix build
ajafff Feb 6, 2021
0e23eba
export new services from public api
ajafff Feb 7, 2021
8444a02
test DependencyResolver
ajafff Feb 7, 2021
ea620aa
Merge branch 'master' into cache
ajafff Feb 7, 2021
b368c54
more testing for dependency resolver
ajafff Feb 7, 2021
c8445f0
add test for ambient module overruling resolved module
ajafff Feb 8, 2021
769afc0
avoid leaking module augmentation for typescript
ajafff Feb 8, 2021
464a868
moar tests for dependency resolver
ajafff Feb 9, 2021
026c5b5
add initial tests for program state
ajafff Feb 9, 2021
4ba0d63
find global augmentation in ambient module
ajafff Feb 10, 2021
1ebf6d6
more tests for program state
ajafff Feb 10, 2021
96b0a80
more testing with circular dependencies
ajafff Feb 10, 2021
a0b8b14
make tests work as intended
ajafff Feb 10, 2021
c7f8f95
Merge branch 'master' of github.com:fimbullinter/wotan into cache
ajafff Feb 11, 2021
e80c556
more circularities
ajafff Feb 11, 2021
5cda856
program state tested, yeah
ajafff Feb 11, 2021
37a13d9
test cache integration in runner
ajafff Feb 11, 2021
4dd2e45
omit case sensitivity from snapshot
ajafff Feb 11, 2021
e67ac5d
make tests typescript version agnostic
ajafff Feb 11, 2021
88d1452
fix import
ajafff Feb 11, 2021
31b26d6
add docs
ajafff Feb 12, 2021
d309c61
Merge branch 'master' into cache
ajafff Feb 12, 2021
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions .dependency-cruiser.json
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,15 @@
"path": "^packages/[^/]+/test/"
}
},
{
"name": "not-to-spec",
"comment": "Don't allow dependencies to spec files",
"severity": "error",
"from": {},
"to": {
"path": "\\.spec\\.[jt]s$"
}
},
{
"name": "not-outside-package",
"comment": "Don't allow packages to anything outside of packages",
Expand Down
3 changes: 3 additions & 0 deletions baselines/packages/wotan/api/index.d.ts
Original file line number Diff line number Diff line change
Expand Up @@ -11,10 +11,13 @@ export * from './src/services/default/line-switches';
export * from './src/services/default/message-handler';
export * from './src/services/default/resolver';
export * from './src/services/default/rule-loader-host';
export * from './src/services/default/state-persistence';
export * from './src/services/cached-file-system';
export * from './src/services/configuration-manager';
export * from './src/services/dependency-resolver';
export * from './src/services/formatter-loader';
export * from './src/services/processor-loader';
export * from './src/services/program-state';
export * from './src/services/rule-loader';
export { parseGlobalOptions, ParsedGlobalOptions, GLOBAL_OPTIONS_SPEC } from './src/argparse';
export * from './src/baseline';
Expand Down
9 changes: 9 additions & 0 deletions baselines/packages/wotan/api/packlist.txt
Original file line number Diff line number Diff line change
Expand Up @@ -93,12 +93,21 @@ src/services/default/resolver.js.map
src/services/default/rule-loader-host.d.ts
src/services/default/rule-loader-host.js
src/services/default/rule-loader-host.js.map
src/services/default/state-persistence.d.ts
src/services/default/state-persistence.js
src/services/default/state-persistence.js.map
src/services/dependency-resolver.d.ts
src/services/dependency-resolver.js
src/services/dependency-resolver.js.map
src/services/formatter-loader.d.ts
src/services/formatter-loader.js
src/services/formatter-loader.js.map
src/services/processor-loader.d.ts
src/services/processor-loader.js
src/services/processor-loader.js.map
src/services/program-state.d.ts
src/services/program-state.js
src/services/program-state.js.map
src/services/rule-loader.d.ts
src/services/rule-loader.js
src/services/rule-loader.js.map
Expand Down
1 change: 1 addition & 0 deletions baselines/packages/wotan/api/src/argparse.d.ts
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ export declare const GLOBAL_OPTIONS_SPEC: {
exclude: OptionParser.ParseFunction<readonly string[]>;
project: OptionParser.ParseFunction<readonly string[]>;
references: OptionParser.ParseFunction<boolean>;
cache: OptionParser.ParseFunction<boolean>;
formatter: OptionParser.ParseFunction<string | undefined>;
fix: OptionParser.ParseFunction<number | boolean>;
extensions: OptionParser.ParseFunction<string[] | undefined>;
Expand Down
4 changes: 3 additions & 1 deletion baselines/packages/wotan/api/src/linter.d.ts
Original file line number Diff line number Diff line change
Expand Up @@ -20,5 +20,7 @@ export declare type UpdateFileCallback = (content: string, range: ts.TextChangeR
export declare class Linter {
constructor(ruleLoader: RuleLoader, logger: MessageHandler, deprecationHandler: DeprecationHandler, filterFactory: FindingFilterFactory);
lintFile(file: ts.SourceFile, config: EffectiveConfiguration, programOrFactory?: ProgramFactory | ts.Program, options?: LinterOptions): ReadonlyArray<Finding>;
lintAndFix(file: ts.SourceFile, content: string, config: EffectiveConfiguration, updateFile: UpdateFileCallback, iterations?: number, programFactory?: ProgramFactory, processor?: AbstractProcessor, options?: LinterOptions): LintAndFixFileResult;
lintAndFix(file: ts.SourceFile, content: string, config: EffectiveConfiguration, updateFile: UpdateFileCallback, iterations?: number, programFactory?: ProgramFactory, processor?: AbstractProcessor, options?: LinterOptions,
/** Initial set of findings from a cache. If provided, the initial linting is skipped and these findings are used for fixing. */
findings?: readonly Finding[]): LintAndFixFileResult;
}
12 changes: 3 additions & 9 deletions baselines/packages/wotan/api/src/runner.d.ts
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
import { Linter } from './linter';
import { LintResult, DirectoryService, MessageHandler, FileFilterFactory, Severity } from '@fimbul/ymir';
import * as ts from 'typescript';
import { ProcessorLoader } from './services/processor-loader';
import { CachedFileSystem } from './services/cached-file-system';
import { ConfigurationManager } from './services/configuration-manager';
import { ProgramStateFactory } from './services/program-state';
export interface LintOptions {
config: string | undefined;
files: ReadonlyArray<string>;
Expand All @@ -13,15 +13,9 @@ export interface LintOptions {
fix: boolean | number;
extensions: ReadonlyArray<string> | undefined;
reportUselessDirectives: Severity | boolean | undefined;
cache: boolean;
}
export declare class Runner {
constructor(fs: CachedFileSystem, configManager: ConfigurationManager, linter: Linter, processorLoader: ProcessorLoader, directories: DirectoryService, logger: MessageHandler, filterFactory: FileFilterFactory);
constructor(fs: CachedFileSystem, configManager: ConfigurationManager, linter: Linter, processorLoader: ProcessorLoader, directories: DirectoryService, logger: MessageHandler, filterFactory: FileFilterFactory, programStateFactory: ProgramStateFactory);
lintCollection(options: LintOptions): LintResult;
}
declare module 'typescript' {
function matchFiles(path: string, extensions: ReadonlyArray<string>, excludes: ReadonlyArray<string> | undefined, includes: ReadonlyArray<string>, useCaseSensitiveFileNames: boolean, currentDirectory: string, depth: number | undefined, getFileSystemEntries: (path: string) => ts.FileSystemEntries, realpath: (path: string) => string): string[];
interface FileSystemEntries {
readonly files: ReadonlyArray<string>;
readonly directories: ReadonlyArray<string>;
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
import { CachedFileSystem } from '../cached-file-system';
import { StatePersistence, StaticProgramState } from '@fimbul/ymir';
export declare class DefaultStatePersistence implements StatePersistence {
constructor(fs: CachedFileSystem);
loadState(project: string): StaticProgramState | undefined;
saveState(project: string, state: StaticProgramState): void;
}
12 changes: 12 additions & 0 deletions baselines/packages/wotan/api/src/services/dependency-resolver.d.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
import * as ts from 'typescript';
export interface DependencyResolver {
update(program: ts.Program, updatedFile: string): void;
getDependencies(fileName: string): ReadonlyMap<string, null | readonly string[]>;
getFilesAffectingGlobalScope(): readonly string[];
}
export declare type DependencyResolverHost = Required<Pick<ts.CompilerHost, 'resolveModuleNames'>> & {
useSourceOfProjectReferenceRedirect?(): boolean;
};
export declare class DependencyResolverFactory {
create(host: DependencyResolverHost, program: ts.Program): DependencyResolver;
}
23 changes: 23 additions & 0 deletions baselines/packages/wotan/api/src/services/program-state.d.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
import * as ts from 'typescript';
import { DependencyResolver, DependencyResolverFactory, DependencyResolverHost } from './dependency-resolver';
import { Finding, StatePersistence } from '@fimbul/ymir';
export interface ProgramState {
update(program: ts.Program, updatedFile: string): void;
getUpToDateResult(fileName: string, configHash: string): readonly Finding[] | undefined;
setFileResult(fileName: string, configHash: string, result: readonly Finding[]): void;
save(): void;
}
export declare class ProgramStateFactory {
constructor(resolverFactory: DependencyResolverFactory, statePersistence: StatePersistence);
create(program: ts.Program, host: ProgramStateHost & DependencyResolverHost, tsconfigPath: string): ProgramStateImpl;
}
export declare type ProgramStateHost = Pick<ts.CompilerHost, 'getCanonicalFileName' | 'useCaseSensitiveFileNames'>;
declare const oldStateSymbol: unique symbol;
declare class ProgramStateImpl implements ProgramState {
constructor(host: ProgramStateHost, program: ts.Program, resolver: DependencyResolver, statePersistence: StatePersistence, project: string);
update(program: ts.Program, updatedFile: string): void;
getUpToDateResult(fileName: string, configHash: string): readonly Finding[] | undefined;
setFileResult(fileName: string, configHash: string, result: ReadonlyArray<Finding>): void;
save(): void;
}
export {};
215 changes: 215 additions & 0 deletions baselines/packages/wotan/test/program-state.spec.ts.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,215 @@
# Snapshot report for `packages/wotan/test/program-state.spec.ts`

The actual snapshot is saved in `program-state.spec.ts.snap`.

Generated by [AVA](https://avajs.dev).

## saves old state

> Snapshot 1

`cs: false␊
files:␊
- config: '1234'␊
hash: '-2704852577'␊
result: []␊
- config: '1234'␊
dependencies:␊
./c:␊
- 0␊
hash: '-5185547329'␊
result: []␊
- dependencies:␊
./c:␊
- 0␊
hash: '-5185547329'␊
- dependencies:␊
./b:␊
- 1␊
./d:␊
- 2␊
hash: '-2126001415'␊
global: []␊
lookup:␊
a.ts: 3␊
b.ts: 1␊
c.ts: 0␊
d.ts: 2␊
options: '-1350339532'␊
v: 1␊
`

> Snapshot 2

`cs: false␊
files:␊
- config: '1234'␊
hash: '-2704852577'␊
result: []␊
- config: '1234'␊
dependencies:␊
./c:␊
- 0␊
hash: '-5185547329'␊
result: []␊
- dependencies:␊
"\\0":␊
- 3␊
hash: '5381'␊
- dependencies:␊
./e:␊
- 2␊
- 3␊
hash: '910822549'␊
- dependencies:␊
./b:␊
- 1␊
./d:␊
- 3␊
hash: '-2126001415'␊
global: []␊
lookup:␊
a.ts: 4␊
b.ts: 1␊
c.ts: 0␊
d.ts: 3␊
e.ts: 2␊
options: '-1350339532'␊
v: 1␊
`

> Snapshot 3

`cs: false␊
files:␊
- hash: '-2704852577'␊
- dependencies:␊
./c:␊
- 0␊
hash: '-5185547329'␊
- dependencies:␊
"\\0":␊
- 3␊
e: null␊
hash: '8844149038'␊
- dependencies:␊
./e:␊
- 2␊
- 3␊
hash: '910822549'␊
- dependencies:␊
./b:␊
- 1␊
./d:␊
- 3␊
hash: '-2126001415'␊
global:␊
- 2␊
lookup:␊
a.ts: 4␊
b.ts: 1␊
c.ts: 0␊
d.ts: 3␊
e.ts: 2␊
options: '-1350339532'␊
v: 1␊
`

## doesn't discard results from old state

> Snapshot 1

`cs: false␊
files:␊
- config: '1234'␊
hash: '-3360789062'␊
result: []␊
- config: '1234'␊
hash: '574235295'␊
result: []␊
- dependencies:␊
./c:␊
- 1␊
hash: '-5185547329'␊
global:␊
- 1␊
lookup:␊
a.ts: 0␊
b.ts: 2␊
c.ts: 1␊
options: '5864093'␊
v: 1␊
`

> Snapshot 2

`cs: false␊
files:␊
- hash: '-3360789062'␊
- hash: '4830905933'␊
- dependencies:␊
./c:␊
- 1␊
hash: '-5185547329'␊
global:␊
- 1␊
lookup:␊
a.ts: 0␊
b.ts: 2␊
c.ts: 1␊
options: '5864093'␊
v: 1␊
`

## handles circular dependencies

> Snapshot 1

`cs: false␊
files:␊
- config: '1234'␊
hash: '-3360789062'␊
result: []␊
- dependencies:␊
./d:␊
- 3␊
hash: '-2335134369'␊
- dependencies:␊
./a:␊
- 4␊
./c:␊
- 1␊
./d:␊
- 3␊
hash: '-10011708814'␊
- dependencies:␊
./b:␊
- 2␊
./e:␊
- 0␊
hash: '970287226'␊
- dependencies:␊
./b:␊
- 2␊
./c:␊
- 1␊
./d:␊
- 3␊
./e:␊
- 0␊
hash: '3028017583'␊
- dependencies:␊
./a:␊
- 4␊
hash: '-5185549507'␊
global: []␊
lookup:␊
a.ts: 4␊
b.ts: 2␊
c.ts: 1␊
d.ts: 3␊
e.ts: 0␊
root.ts: 5␊
options: '5864093'␊
v: 1␊
`
Binary file not shown.
Loading