1. 25 Apr, 2017 1 commit
    • huangs's avatar
      [Courgette] Refactor: Add CourgetteFlow; improve courgette_tool.cc help text. · 8b91a4c9
      huangs authored
      High level flows in Courgette has messy error-handling code that's
      hard to read. This CL adds CourgetteFlow to absorb commonly used data
      and error-handling code, and improve readability of high-level logic.
      Additional changes:
      - Fix Courgette-disadj (debugging flow), which likely was broken by
        http://crrev.com/2793153003; Adjust() is called but AssemblyProgram
        did not annotate Labels.
      - Add BasicBuffer interface, implemented by RegionBuffer and (existing)
        BufferedFileReader.
      - PatchGeneratorX86_32::Transform(): Implemented TODO to instantiate
        "new" AssemblyProgram to reduce Courgette-gen peak memory.
      - courgette_tools.cc: Make help text more uniform.
      
      BUG=660980
      
      Review-Url: https://codereview.chromium.org/2827103002
      Cr-Commit-Position: refs/heads/master@{#467157}
      8b91a4c9
  2. 13 Apr, 2017 1 commit
    • huangs's avatar
      [Courgette] Refactor: Store Label Annotation in AssemblyProgram for patch generation. · c4155eb6
      huangs authored
      For Courgette-gen, label adjustment needs lists of abs32 & rel32 Label*
      sorted by file offset. Let these lists be "Label Annotations".
      Previously these were extracted during label adjustment from list of
      instructions in AssemblyProgram, but now we wish to remove these stored
      instructions.
      
      This CL make AssemblyProgram store Label annotations. These are computed
      only when needed (Courgette-gen / -gen1a). Details:
      
      - Add ParseDetecteExecutableWithAnnotation() alongside
        ParseDetecteExecutable(), to avoid affecting flows that don't require
        Label Annotations.
      - AssemblyProgram: Add |*_label_annotations_| as storage vectors. These
        are optionally populated in InstructionStoreReceptor when
        Disassembler::Disassemble() gets called.
        - InstructionCountReceptor now put into use.
      - Simplify Label adjustment: AssemblyProgram::HandleInstructionLabels()
        is replaced with visiting AssemblyProgram's Label Annotations.
      - Subtle: GraphAdjuster now includes abs64. The class doesn't get used,
        but the change is logical anyway.
      
      BUG=660980
      
      Review-Url: https://codereview.chromium.org/2793153003
      Cr-Commit-Position: refs/heads/master@{#464536}
      c4155eb6
  3. 01 Apr, 2017 1 commit
  4. 24 Mar, 2017 1 commit
  5. 23 Mar, 2017 1 commit
    • huangs's avatar
      [Courgette] Refactor: Unify Disassembler::Disassemble() and instantiate AssemblyProgram there. · 257f9fb0
      huangs authored
      This is part of AssemblyProgram reduction effort. Details:
      - Add Disassembler::Disassemble() and dedup separate PE/ELF versions.
        - Instantiate AssemblyProgram there; replaces old behavior where
          caller (program_detector.cc) instantiates and passes as out param.
      - Add virtual Disassembler::GetInstructionGenerator().
        - Remove InstructionGenerator's AssemblyProgram* param; update tests.
      - Move InstructionReceptor and InstructionGenerator from AssemblyProgram
        to new file instruction_utils.h.
      - Rename ParseAbs32Relocs() to ExtractAbs32Locations(); make common.
      - Rename ParseRel32Relocs() to ExtractRel32Locations(); make common.
      - Make DisassemblerElf32::abs32_locations_ non-mutable.
      
      BUG=660980
      
      Review-Url: https://codereview.chromium.org/2771753004
      Cr-Commit-Position: refs/heads/master@{#459272}
      257f9fb0
  6. 22 Mar, 2017 1 commit
    • huangs's avatar
      [Courgette] Fix ELF reference sorting. · c615c911
      huangs authored
      This CL addresses 2 reference sorting issues in DisassemblerElf32:
      
      (1) Bug fix: In ParseFile(), |abs32_locations_| (RVAs) is translated to
          |abs_offsets| (file offsets), but we sort |abs32_locations_|, which
          is redundant. Actually we should sort |abs_offsets|.
      (2) Cleanup: |rel32_relocations_| stores rel32 references sorted by
          RVA, but in ParseFile() we re-sort these in offset order. Previously
          Disassemble() optimizes away redundant sorts, but this makes the
          code less robust. We de-optimize this a little potentially redundant
          sort-by-RVA, to assert that |rel32_locations_| is sorted by RVA
          outside of ParseFile().
      
      This CL also makes Disassemble() more uniform, to prepare for
      refactoring in a follow-up. Meanwhile, DisassemblerWin32 does not
      experience issue since it assumes RVA order is same as file offset
      order (this assumption has not has not caused problems so far).
      
      BUG=660980
      
      Review-Url: https://codereview.chromium.org/2744373004
      Cr-Commit-Position: refs/heads/master@{#458650}
      c615c911
  7. 20 Mar, 2017 1 commit
  8. 01 Mar, 2017 1 commit
    • huangs's avatar
      [Courgette] Disassembler code movement and format fix. · dc779d9f
      huangs authored
      We wish to rename ParseAbs32Relocs() and ParseRel32RelocsFromSections(),
      and make them part of the Disassembler interface. This requires moving
      chunks of code. This CL does the code movement as an independent change
      to reduce noise later.
      
      Also including reformatting changes from "git cl format".
      
      BUG=660980
      
      Review-Url: https://codereview.chromium.org/2728653003
      Cr-Commit-Position: refs/heads/master@{#454068}
      dc779d9f
  9. 15 Feb, 2017 1 commit
  10. 13 Jan, 2017 1 commit
    • huangs's avatar
      [Courgette] Simple AssemblyProgram and Disassembler cleanups. · 85636222
      huangs authored
      We perform mundane cleanups to prepare for next step in
      AssemblyProgram reduction:
      - Delete dead code (static functions) in AssemblyProgram that were left
        over from LabelManager usage.
      - Simplify image_base assignment for AssemblyProgram.
        - Source: Disassembler adds virtual function image_base() (PE file
          reads and stores it; ELF just returns 0).
        - Sink: AssemblyProgram takes it on construction (replaces mutator).
      - Disassembler::Disassemble() overrides: Rename |target| to |program|.
      
      BUG=660980
      
      Review-Url: https://codereview.chromium.org/2583373002
      Cr-Commit-Position: refs/heads/master@{#443609}
      85636222
  11. 09 Dec, 2016 1 commit
    • waffles's avatar
      Sandbox the component updater's patcher utility process. · 53796c72
      waffles authored
      The code will now open files in the browser process before passing the
      handles across IPC to the utility process. The utility process in turn
      invokes courgette/bsdiff, which memory maps the files and operates on
      them as before.
      
      There is a behavioral difference when using the courgette or
      courgette_mini tools: the output file will now be created/overwritten
      at the start of the operation, and in the case of a failure, will be
      deleted. Previously, the output file was created late in the operation
      operation and several failure modes would leave it unmodified.
      
      BUG=660325
      
      Review-Url: https://codereview.chromium.org/2534873005
      Cr-Commit-Position: refs/heads/master@{#437669}
      53796c72
  12. 07 Dec, 2016 1 commit
    • jbriance's avatar
      Cleanup remaining class/struct fwd declarations · db24b9e6
      jbriance authored
      Remove remaining useless class and struct forward declarations from
      Chromium (excluding third_party directory) using a homemade script
      available in bug 662195 comments. An up-and-running presubmit check
      is now available and should prevent new useless forward declarations
      to come up.
      
      BUG=662195
      CQ_INCLUDE_TRYBOTS=master.tryserver.blink:linux_trusty_blink_rel;master.tryserver.chromium.linux:linux_nacl_sdk
      
      Review-Url: https://codereview.chromium.org/2554863002
      Cr-Commit-Position: refs/heads/master@{#436883}
      db24b9e6
  13. 30 Nov, 2016 1 commit
  14. 22 Nov, 2016 1 commit
  15. 09 Nov, 2016 1 commit
    • huangs's avatar
      [Courgette] Refactor: Add and use Instruction*Receptor classes; call ParseFile() in 2 passes. · 7b221a5e
      huangs authored
      We wish to reduce AssemblyProgram's memory usage on instruction storage.
      This requires changing Disassembler*::Disassemble() to call ParseFile()
      in 2 passes: (1) count bytes needed (so we can preallocate) (2) store
      bytes. This CL implements the 2-pass system, focusing on callers. Details:
      - Add AssemblyProgram::GenerateInstructions() as main entrance point
        from Disassembler*. It takes InstructionGenerator, which is just
        Disassembler*::ParseFile() wrapped in a Closure.
      - Change Disassembler*::ParseFile() and downstream functions to const,
        to assert absence of Disassembler state change for the passes.
        - Exception: For ELF, we need to stort abs32_locations_ and
          re32_locations_ to obtain different traversal orders. This doesn't
          add/remove elements. So we make these containers mutable.
      - Emit*() functions are moved from AssemblyProgram to new interface
        InstructionReceptor. Updating all Emit*() calls to accommodate.
      - Add InstructionReceptor interface with 2 implementations:
        - InstructionCountReceptor for pass 1 (no-op in this CL).
        - InstructionStoreReceptor for pass 2 (calls AssemblyProgram::Emit*()
          in this CL).
      - Rename Emit*Instruction() to Emit*().
      
      TEST=Correctness without behavioral change; Courgette-gen and Courgette-apply are not slowed down, and do not use more memory.
      BUG=660980
      
      Review-Url: https://codereview.chromium.org/2462993003
      Cr-Commit-Position: refs/heads/master@{#431053}
      7b221a5e
  16. 28 Oct, 2016 1 commit
    • huangs's avatar
      [Courgette] Refactor: Add AssemblyProgram::DispatchInstructionLabels() to hide... · 99a5a8c3
      huangs authored
      [Courgette] Refactor: Add AssemblyProgram::DispatchInstructionLabels() to hide InstructionVector usage.
      
      To reduce Courgette memory usage (and possibly solve installer crashes),
      we plan to make AssemblyProgram store instructions more efficiently. We
      start by hiding AssemblyProgram's InstructionVector usage from
      AdjustmentMethod and AdjustmentMethod2.
      
      Previously AdjustmentMethod[2] get AssemblyProgram's instruction vector,
      and loop over it to extract labels from abs32/rel32 instructions. This
      CL moves the loop into AssemblyProgram::DispatchInstructionLabels().
      Callers now specify a map of handlers, without needing to know how
      instructions are stored in AssemblyProgram.
      
      Review-Url: https://codereview.chromium.org/2457133002
      Cr-Commit-Position: refs/heads/master@{#428522}
      99a5a8c3
  17. 14 Sep, 2016 1 commit
    • brettw's avatar
      Remove GYP files. · 315e4cfd
      brettw authored
      This removes all .gyp files not called "compiled_resources" in the main Chromium repository.
      CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.win:win10_chromium_x64_rel_ng
      
      Review-Url: https://codereview.chromium.org/2332843004
      Cr-Commit-Position: refs/heads/master@{#418654}
      315e4cfd
  18. 01 Sep, 2016 1 commit
    • maruel's avatar
      Delete all test isolate file now that M54 was branched. · 95b6282d
      maruel authored
      The runtime dependencies data was successfully transitioned to GN.
      
      Keep non-testing isolate as some projects (v8, webrtc) still need some. They
      will be curated incrementally.
      
      Ran:
        $ git rm $(git ls-files -- "*.isolate" | grep -i test | grep -v testserver)
        $ git ls-files -- "*.isolate" | wc -l
        14
      
      TBR=dpranke@chromium.org
      BUG=
      CQ_INCLUDE_TRYBOTS=master.tryserver.blink:linux_precise_blink_rel
      
      Review-Url: https://codereview.chromium.org/2257893002
      Cr-Commit-Position: refs/heads/master@{#415954}
      95b6282d
  19. 02 Aug, 2016 1 commit
  20. 30 Jul, 2016 1 commit
    • etienneb's avatar
      Fix GN generation for WinASAN build · 364deb71
      etienneb authored
      This is a step toward a working Asan build with GN.
      There are still a few missing pieces to land.
      
      This patch is fixing an assert triggering due to courgette using an
      harcoded toolchains.
      
      ----------------------
       ERROR at //build/config/sanitizers/sanitizers.gni:12:1: Assertion failed.
        assert(!using_sanitizer || is_clang,
        ^-----
        Sanitizers (is_*san) require setting is_clang = true in 'gn args'
      ----------------------
      
      This patch is also fixing the compiler-rt runtime paths for a x64 bit
      build. The port of WinASAN-64 is almost done and should part of the next
      clang roll-deps.
      
      R=thakis@chromium.org, dpranke@chromium.org
      TBR=brettw@chromium.org
      BUG=598761
      
      Review-Url: https://codereview.chromium.org/2192833002
      Cr-Commit-Position: refs/heads/master@{#408857}
      364deb71
  21. 27 Jul, 2016 2 commits
  22. 26 Jul, 2016 1 commit
    • huangs's avatar
      [Courgette] Refactor BSDiff namespaces and bsdiff::search() interface. · 7054b5a2
      huangs authored
      Details:
      - Move BSDiff (but not PagedArray) from namespace courgette to bsdiff.
      - Change namespace courgette::qsuf to qsuf.
      - Change bsdiff:search() to return struct {pos, size} so we don't need
        awkward pointer passing; update callers.
      - Updated BSDiff callers. Also fix weird hybrid usage by setup_util.cc,
        which calls Courgette's BSDiff, but using OK from BSPatch.
      
      BUG=608885
      
      Review-Url: https://codereview.chromium.org/2031193002
      Cr-Commit-Position: refs/heads/master@{#407924}
      7054b5a2
  23. 20 Jul, 2016 1 commit
  24. 19 Jul, 2016 1 commit
  25. 14 Jul, 2016 1 commit
    • etiennep's avatar
      Use memory mapped file in courgette_tool · f4fe06cf
      etiennep authored
      Patch generation and various utilities (disasm) now use memory mapped file
      instead of copying into a string buffer. This saves the equivalent of the size of
      both |old_file| and |new_file| in memory when generating the patch
      (~300Mb with chrome.7z).
      
      BUG=
      
      Review-Url: https://codereview.chromium.org/2143973004
      Cr-Commit-Position: refs/heads/master@{#405599}
      f4fe06cf
  26. 13 Jul, 2016 1 commit
    • etiennep's avatar
      Refactor courgette_tool when loading files. · d10aa3ea
      etiennep authored
      The intention is to replace usages of char* by uint8_t* in DetectExecutableType.
      For this, the function ReadOrFile is replaced by a class
      BufferedFileReader, which provide an accessor on the data.
      This will allow us to easily investigate on using MemoryMappedFile.
      
      BUG=
      
      Review-Url: https://codereview.chromium.org/2140203002
      Cr-Commit-Position: refs/heads/master@{#405333}
      d10aa3ea
  27. 08 Jul, 2016 1 commit
    • etiennep's avatar
      Courgette: Add static method QuickDetect() to optimize program detection. · 5059bca6
      etiennep authored
      Old way: The detection of executables is achieve by allocating an instance of each
      Disassembler classes and trying to call ParseHeader() to see if it succeed.
      This operation is done many times during FindEmbeddedElements() step,
      which takes ~4% of patch generation time.
      New way: Using QuickDetect(), which executes only a quick preliminary check,
      we avoid useless allocation of Disassembler objects unless
      there's a high probability of successfully detecting a valid executable.
      This change reduces the execution time of FindEmbeddedElements() by 95%.
      
      BUG=619167
      
      Review-Url: https://codereview.chromium.org/2055343002
      Cr-Commit-Position: refs/heads/master@{#404433}
      5059bca6
  28. 28 Jun, 2016 2 commits
  29. 15 Jun, 2016 1 commit
  30. 07 Jun, 2016 1 commit
  31. 03 Jun, 2016 1 commit
    • thakis's avatar
      asan/win: Unbreak build after https://codereview.chromium.org/2008553007/ · d36b726b
      thakis authored
      ..\..\courgette\third_party\bsdiff\paged_array_unittest.cc(20,11):
        error: unused variable 'kIterations' [-Werror,-Wunused-const-variable]
      const int kIterations = 20;
                ^
      ..\..\courgette\third_party\bsdiff\paged_array_unittest.cc(21,11):
        error: unused variable 'kSizeBig' [-Werror,-Wunused-const-variable]
      const int kSizeBig = 200 * 1024 * 1024 / sizeof(int);  // 200MB
                ^
      
      BUG=82385
      TBR=huangs
      
      Review-Url: https://codereview.chromium.org/2036393002
      Cr-Commit-Position: refs/heads/master@{#397810}
      d36b726b
  32. 02 Jun, 2016 1 commit
    • huangs's avatar
      [Courgette] PagedArray: Add Iterators and Parametrize Page Size as int Template. · 804ed8a1
      huangs authored
      This is a refactoring CL to enable PagedArray usage by libdivsufsort.
      In addition to overloading operator[], for more general usage we need
      need pointer-like accessors to PagedArray. To this end we implement
      PagedArray_const_iterator and PagedArray_const_iterator, which merely
      wraps a PagedArray pointer along with an index. We also add various
      operators needed by libdivsufsort. For optimization, '<' and '<='
      operators omits pointer checks.
      
      By default PagedArray page size is 2**18 elements (1 MiB for int32_t).
      To enable better testing, we made (log) page size a tepmlate parameter.
      
      BUG=608885
      
      Review-Url: https://codereview.chromium.org/2008553007
      Cr-Commit-Position: refs/heads/master@{#397311}
      804ed8a1
  33. 24 May, 2016 1 commit
    • sdefresne's avatar
      [iOS/GN] Fix the compilation of "all" on iOS with gn. · 6fd373e5
      sdefresne authored
      Move definition of executables that cannot be run on iOS, sources
      that do not build, test suite that are empty and unsupported deps
      behind !is_ios checks.
      
      Add "assert(!is_ios)" in some WebKit BUILD.gn as it cannot be used
      on iOS (need to use the version shipped with the system).
      
      This fix running ninja without specifying any target (i.e. building
      "all").
      
      BUG=603127,459705
      
      Review-Url: https://codereview.chromium.org/2004743002
      Cr-Commit-Position: refs/heads/master@{#395699}
      6fd373e5
  34. 20 May, 2016 1 commit
  35. 19 May, 2016 1 commit
    • huangs's avatar
      [Courgette] Using LabelManager to reduce Courgette-apply peak RAM by 25%. · c803763b
      huangs authored
      AssemblyProgram previously allocates new Label instances as it parses
      an executable and emits instructions. This CL replaces the flow by using
      LabelManager to precompute Labels in one array. This allows us to reduce
      Courgette-apply peak RAM by 25%, measured by "choke RAM until failure"
      method. Details:
      - We precompute Labels in AssemblyProgram::PrecomputeLabels(), which
        relies on RvaVisitor inherited classes for architecture-specific
        extraction of abs32 and rel32 targets.
      - TrimLabel()'s complex post-processing flow is simplified using
        PrecomputeLabels(), which runs before main file parse.
        - This requires RemoveUnusedRel32Locations() to update rel32.
        - Deprecating C_TRIM_FAILED error message.
      - Moving more common functionality to Disassembler, but duplicating
        some code for win32-x86 and win32-x64 to follow existing pattern.
      
      BUG=613216
      
      Review-Url: https://codereview.chromium.org/1935203002
      Cr-Commit-Position: refs/heads/master@{#394815}
      c803763b
  36. 18 May, 2016 1 commit
    • altimin's avatar
      Move //courgette/third_party to subfolder. · 979ea2e1
      altimin authored
      Move //courgette/third_party/ to //courgette/third_party/bsdiff
      in order to conform to Chromium style of putting third-party
      libraries to named directories.
      
      Also fix some warnings about wrong include order.
      
      R=wfh@chromium.org,huangs@chromium.org
      CC=alexclarke@chromium.org,skyostil@chromium.org
      
      Review-Url: https://codereview.chromium.org/1961963003
      Cr-Commit-Position: refs/heads/master@{#394437}
      979ea2e1
  37. 11 May, 2016 1 commit
  38. 28 Apr, 2016 1 commit
    • huangs's avatar
      [Courgette] ELF: Fix abs32 / rel32 ordering in ParseFile() and restrict rel32 parsing to .text. · 3da0dd93
      huangs authored
      This CL fixes 2 problems in Courgette ELF parsing:
      1. ParseFile() scans through file bytes in file offset order while
         visiting abs32 and rel32 locations in lockstep. However, these
         locations were previously not sorted by file offset, resulting in
         some abs32 and rel32 locations being ignored.
      2. ParseRel32RelocsFromSections() is too permissive, and extracts bogus
         rel32 addresses from non-code.
      
      To solve (1) we sort abs32 and rel32 addresses by file offset in
      ParseFile(). To solve (2) we restrict rel32 parsing to ".text" section
      (heuristic). Also updating ELF test results.
      
      BUG=601948
      
      Review-Url: https://codereview.chromium.org/1928683002
      Cr-Commit-Position: refs/heads/master@{#390506}
      3da0dd93