Conventional vision systems suffer from lots of data handling between memory and processing units. Inspired by how humans recognize noisy images and the flexible modulation on the timescale of ion dynamics inside an emerging memtransistor, we report a novel neuromorphic vision system based on the ion-modulated memtransistors. By controlling the ion doping processes under adequate stimuli strengths, both short-term and long-term ion dynamics can be utilized to deliver energy-efficient data processing. When dealing with image reconstructions, the short-term accumulation effect of the device can help filter noises in a set of received noisy images while enhancing the original pattern information. The increased contrast can help distinguish the actual contents. To demonstrate systematic performances with the reconfiguration of devices, we extract the nonlinear relationship between channel conductance variation and the amplitude of gate pulses into the network-level simulation. Also, with the nonvolatile conductance change characteristic, the task of recognizing noisy images is performed to verify the versatility of ion-modulated memtransistors in the neuromorphic artificial vision systems.

Manipulation strategies based on the passive dynamics of soft-bodied interactions provide robust performances with limited sensory information. They utilise the kinematic structure and passive dynamics of the body to adapt to objects of varying shapes and properties. However, these soft passive interactions make the state of the robotic device influenced by the environment, making control generation and state estimation difficult. This work presents a closed-loop framework for dynamic interaction-based grasping that relies on two novelties: (i) a wrist-driven passive soft anthropomorphic hand that can generate robust grasp strategies using one-step kinaesthetic teaching and (ii) a learning-based perception system that uses temporal data from sparse tactile sensors to predict and adapt to failures before it happens. With the anthropomorphic soft design and wrist-driven control, we show that controllers can be generated robust to novel objects and location uncertainty. With the learning-based high-level perception system and 32 sensing receptors, we show that failures can be predicted in advance, further improving the robustness of the entire system by more than doubling the grasping success rate. From over 1000 real-world grasping trials, both the control and perception framework are also seen to be transferable to novel objects and conditions. Corresponding author(s) Email:  _ fi224@cam.ac.uk  _

Aerial robots can autonomously collect temporal and spatial high-resolution environmental data. This data can then be utilized to develop mathematical ecology models to understand the impact of climate change on our habitat. In case of the drone's malfunction the incorporated materials can threaten vulnerable environments. The recent introduction of transient robotics has enabled the development of biodegradable, environmental sensing drones capable of degrading in their environment. However, manufacturing methods for environmental sensing transient drones are rarely discussed. In this work, we highlight a manufacturing framework and material selection process featuring biopolymer-based, high-strength composite cryogels and printed carbon-based electronics for transient drones. We found that gelatin and cellulose based cryogels mechanically outperform other biopolymer composites while having a homogeneous micro-structure and high stiffness-to-weight ratio. The selected materials are used to manufacture a flying-wing air-frame, while the incorporated sensing skin is capable of measuring the elevons' deflection angles as well as ambient temperature. Our results demonstrate how gelatin-cellulose cryogels can be used to manufacture lightweight transient drones while printing carbon conductive electronics is a viable method for designing sustainable, integrated sensors. The proposed methods can be used to guide the development of lightweight and rapidly degrading robots, featuring eco-friendly sensing capabilities.Corresponding author(s) Email:   fabian.wiesemueller@empa.ch, mirko.kovac@empa.ch 

Artificial muscles with large strokes are of special interest in diverse fields. However, it is difficult for large-diameter muscles to be rapidly cycled. In this study, hair artificial muscles with extremely large tensile stroke and fast recovery were prepared simply by twist insertion, coiling and steaming. The maximum tensile stroke for the hair artificial muscles upon water actuation was as large as 10000% and the large-stroke muscles could recover fast in ethanol. With a diameter of 7 mm and a twist density of 2500 turns m-1, the compacted heterochiral hair artificial muscle could elongate 100 times of its original length in water and returned to its initial length in ethanol within 10 s. In addition, these hair artificial muscles maintained their excellent performance after either 100 water-ethanol stimulation cycles or staying in open air for 5 months. Moreover, the hair artificial muscle was able to contract by 59% when lifting 10 times its own weight, pull a wheel model or climb a long distance under water and work as a smart water-sensitive switch. This work demonstrates a facile and green strategy to prepare advanced natural fiber-based artificial muscles that have promising applications in soft robotics and biomedical engineering. Corresponding author(s) Email:  sunsi@gzhu.edu.cn (Dr. Si Sun), qiangxl@gzhu.edu.cn (Dr. Xiao-Li Qiang), & xlshi@gzhu.edu.cn (Dr. Xiao-Long Shi)

The theoretical capability of modular robot to organize the overall robot into different structures with different functions has broad prospects in space exploration. Therefore, we develop a novel modular space robot named Space Module, and inspired by biological cooperative and mutual assistance behaviors, a novel self-assembly method is proposed for it.To solve the mobility problem of non-mobile modules, a new meta-modules design for Space Module is presented, based on which the concept of mutual assistance is utilized to achieve position and posture reachability of assembled unit while minimizing the effect of meta-modules on granularity. Then, an assembly planner is designed to obtain the assembly sequences according to the unique motion characteristics of meta-module and mutual assistance to realize the self-manufacturing of desired configurations. Finally, several demonstrations are given to verify the validity and feasibility of the proposed assembly method.Corresponding author(s) Email:   yhzhu@hit.edu.cn

With advancements in automation and high-throughput techniques, complex materials discovery with multiple conflicting objectives can now be tackled in experimental labs. Given that physical experimentation is greatly limited by evaluation budget, maximizing efficiency of optimization becomes crucial. We discuss the limitations of using hypervolume as a performance indicator for desired optimality across the entire multi-objective optimization run and propose new metrics specific to experimentation: ability to perform well for complex high-dimensional problems, minimizing wastage of evaluations, consistency/robustness of optimization, and ability to scale well to high throughputs. With these metrics, we perform a comparison of two conceptually different and state-of-the-art algorithms (Bayesian and Evolutionary) on synthetic and real-world datasets. We discuss the merits of both approaches with respect to exploration and exploitation, where fully resolving the Pareto Front could be the main aim for greater scientific value in understanding materials space, and thus provide a perspective for materials scientists to implement optimization in their platforms.

The metaverse, where the virtual and real world are fused, is currently under rapid development. Immersive and vivid experience in the metaverse requires human-machine interaction devices that, unlike those currently available, are simultaneously imperceptible, convenient to use, inexpensive, and safe. In this study, we propose and realize an optical-nanofiber-based gesture-recognition wristband that can accurately recognize gestures and use them to interact with a robotic hand. Requiring only three optical-nanofiber-based pressure sensors, the wristband is simple in structure, convenient to use, and remarkably imperceptible to the user. With the assistance of a machine-learning algorithm, a maximum recognition accuracy of 94% is achieved for testers with different physiques. A robotic hand can be remotely controlled by the wristband through gestures. The wristband has broad application prospects and is a promising solution for advanced human-machine-interaction devices.

AbstractIn recent years, there has been a growing interest in the development of universal soft grippers that can handle objects of varying form factors (including flat objects), surface condition (including moistened or oily objects), and mechanical properties (deformable and fragile). Yet, there is no single gripper that can gently grip objects with such a wide range of properties. In this paper, we present a soft gripper that combines granular jamming (GJ) and electroadhesion (EA) to gently grasp and release a large set of diverse objects. The gripper can operate in GJ mode only, in EA mode only, or in a combination mode that simultaneously activates GJ and EA. In GJ mode, the gripper can grasp objects with different surface properties, lift objects 38 times its own weight using negative pressure, and release objects by applying positive pressure, but has difficulty in handling flat and fragile objects. In EA mode, the gripper can manipulate flat and fragile objects but encounters difficulties with different surface properties such as oily or moistened. In the combination mode, the gripper can generates grasping forces up to 35% higher than in the GJ mode for all object sizes and certain shapes such as a cylinder.IntroductionThe softness of the human hand is a critical factor that allows us to hold, lift, and manipulate a variety of objects and has inspired roboticists to incorporate softness in gripper design and materials. The compliance of soft materials enables passive adaptation of the gripper during grasping operations allowing manipulation of a wide range of objects without bringing additional control complexities.[1,2] In recent years, there has been a growing interest in the development of universal soft grippers that can work with objects of different form factors, rigidity, surface properties, and level of fragility.[3–6] A possible approach to create such highly versatile grippers is to combine different gripping technologies that complement their individual limitations.[1,3,7–10] Yet, it is still challenging to develop a single gripper that can grasp and release objects of different form factor including flat objects, surface conditions (wet, porous, oily, and powdered), and mechanical properties (fragile and deformable).In this paper, we present a soft gripper capable of manipulating different objects with varying physical properties, such as shape, surface conditions, and rigidity. The proposed gripper combines two different technologies: granular jamming to control stiffness and electroadhesion to control adhesion. Here we show that not only does this combination mutually compensate for the limitations of each individual technology, but it also makes the gripper capable of performing multi-stage grasping tasks that consist of diverse grasping and releasing operations on objects made of different material, surface, and shape. The manipulation of a book is an example of multi-stage operation that requires grasping and turning a rigid cover and flipping through single pages.Granular jamming (GJ) enables reversible stiffness change between soft and rigid configurations by means of negative pressure[5,11,12]. High compliance in the soft state allows a GJ gripper to envelope the manipulated object by pressing on it. When negative pressure is applied, the gripper becomes stiff and holds the encaged object.[2] Variable stiffness can also be achieved by integrating phase-change materials that vary mechanical properties under thermal stimulation.[7,13] However, granular jamming offers comparatively faster response time (~100ms), independence from environmental temperature, higher lifting force, easier fabrication, higher robustness, and lower cost.[2,5,14,15] The grasping force produced by granular jamming is sufficient to grasp objects of different morphologies, almost independently of the surface conditions of the object.[5,11,16] The grasping force of GJ grippers can vary from 0.09 to 1.2 kN.[12] GJ has been combined with soft pneumatic actuators to provide more dexterous grasp and lift heavier objects because of the enhanced holding forces.[10,17] However, GJ grippers cannot lift flat objects, such as a sheet of paper. Also, the grasping performance of delicate, fragile and easily deformable objects such as a thin layer of cloth, an egg, or water balloons, as well as larger objects than the active area of the granular bag can be challenging and have not been demonstrated so far.Electroadhesion (EA) instead is an adhesive technology that leverages the shear force generated by electrostatic forces.[18] Electroadhesive pads have been combined with different actuation technologies, such as dielectric elastomer actuation,[1] soft pneumatic actuation,[19] layer jamming,[20] and Fin-Ray structured actuation[21]. The enhanced shear force makes EA-based grippers capable of delicately grasping both flat and fragile objects without squeezing or breaking them.[1,22–26] While the adhesive force of EA pads can be tuned by regulating electrical input, EA effectiveness is highly dependent on the environmental and surface conditions of the object being grasped.[18] In particular electroadhesion is less effective for objects that are greasy, rough, or wet.[2] An additional challenge of soft grippers that rely on electroadhesion is the residual electrostatic charge that remains for a few seconds after removing the voltage and can result in difficult release of light objects.[27]

Multi-chamber soft pneumatic actuators (m-SPAs) have been widely used in soft robotic systems to achieve versatile grasping and locomotion. However, existing m-SPAs have slow actuation speed and are either limited by a finite air supply or require energy-consuming hardware to continuously supply compressed air. Here, we address these shortcomings by introducing an internal exhaust air recirculation (IEAR) mechanism for high-speed and low-energy actuation of m-SPAs. This mechanism recirculates the exhaust compressed air and recovers the energy by harnessing the rhythmic actuation of multiple chambers. We develop a theoretical model to guide the analysis of the IEAR mechanism, which agrees well with the experimental results. Comparative experimental results of several sets of m-SPAs show that our IEAR mechanism significantly improves the actuation speed by more than 82.4% and reduces the energy consumption per cycle by more than 47.7% under typical conditions. We further demonstrate the promising applications of the IEAR mechanism in various pneumatic soft machines and robots such as a robotic fin, fabric-based finger, and quadruped robot. Corresponding author(s) Email:    guguoying@sjtu.edu.cn

Atomic force microscopy (AFM) is routinely used as a metrological tool among diverse scientific and engineering disciplines. A typical AFM, however, is intrinsically limited by low throughput and is inoperable under extreme conditions. Thus, this work attempts to provide an alternative with a conventional optical microscope (OM) by training a deep learning model to predict surface topography from surface OM images. The feasibility of our novel methodology is shown with germanium-on-nothing (GON) samples, which are self-assembled structures that undergo surface and sub-surface morphological transformations upon high-temperature annealing. Their transformed surface topographies are predicted based on OM-AFM correlation of 3 different surfaces, bearing an error of about 15% with 1.72× resolution upscale from OM to AFM. The OM-based approach brings about significant improvement in topography measurement throughput (equivalent to OM acquisition rate, up to 200 frames per second) and area (∼1 mm²). Furthermore, this method is operable even under extreme environments when an _in-situ_ measurement is impossible. Based on such competence, we also demonstrate the model’s simultaneous application in further specimen analysis, namely surface morphological classification and simulation of dynamic surfaces’ transformation.

Organic memristors are promising candidates for the flexible synaptic components of wearable intelligent systems. With heightened concerns for the environment, considerable effort has been made to develop organic transient memristors to realize eco-friendly flexible neural networks. However, in the transient neural networks, achieving flexible memristors with bio-realistic synaptic plasticity for energy efficient learning processes is still challenging. Here, we demonstrate a biodegradable and flexible polymer based memristor, suitable for the spike-dependent learning process. An electrochemical metallization phenomenon for the conductive nanofilament growth in a polymer medium of poly (vinyl alcohol) (PVA) is analyzed and a PVA based transient and flexible artificial synapse is developed. The developed device exhibits superior biodegradability and stable mechanical flexibility due to the high water solubility and excellent tensile strength of the PVA film, respectively. In addition, the developed flexible memristor is operated as a reliable synaptic device with optimized synaptic plasticity, which is ideal for artificial neural networks with the spike-dependent operations. The developed device is found to be effectively served as a reliable synaptic component with high energy efficiency in practical neural networks. This novel strategy for developing transient and flexible artificial synapses can be a fundamental platform for realizing eco-friendly wearable intelligent systems.Corresponding author(s) Email:   sinhlee@knu.ac.kr 

IntroductionReal world Time on Treatment (rwToT), also known as real world time to treatment discontinuation (rwTTD), is defined as the length of time observed in real world data (as distinct from controlled clinical trials) from initiation of a medication to discontinuation of that medication1,2. The ending of the treatment can be caused by adverse events, deaths, switches of treatment and loss of follow up. Because time to treatment discontinuation can be readily obtained from electronic medical records, this effectiveness endpoint is convenient to evaluate the efficacy of a drug that is already approved for public use3. It is often used as a surrogate effectiveness endpoint, showing high correlation to progression-free survival and moderate-to-high correlation to overall survival4,5. As rwTTD is an important metric for drug effectiveness, it is routinely reported during the post-clinical trial phase2,4,6–9.Calculation of rwTTD in patient population is often equivalent to constructing a (Kaplan-Meier) KM curve, with each point representing the proportion of patients that are still on treatment at a specific time point 1. Either the entire curve, or mean rwTTD, restricted mean10, or the time point at which a specific portion of the patients (e.g. , 50%) dropping treatment is of interest. Currently, there is no existing machine learning scheme established to predict such a curve, or the midpoint, as the vast majority of the machine learning models have been focused on predicting individuals’ behavior rather than population-level behavior. Such a machine learning scheme, if established, has many meaningful clinical applications. For instance, given observed clinical parameters and outcomes in clinical trials, how do we derive expected time-to-treatment in the real-world? Given the rwTTD for a drug on one patient population, how can we predict the rwTTD when applying this drug to another population (e.g. , for a different disease)?This study establishes a machine learning framework to infer population-wise rwTTD. We showed that population-wise curve prediction differs substantially from aggregating all individuals’ results. Our framework models the population-wise curve and is generic to diverse base-learners for predicting rwTTD. We demonstrated the effectiveness of this framework based on both simulated data and real world Electronic medical records (EMR) data for pembrolizumab-treated cancer populations7,11,12. The study opens a new direction of modeling population-level rwTTD, which has great values for directing post-clinical stage drug administrations. This machine learning scheme will also have meaningful implications to population-based predictions for other problems, as machine learning algorithms have so far been focused on predictions for individual samples.

Multiparametric magnetic resonance imaging (mpMRI) has emerged as a valuable diagnostic tool in prostate lesion assessment, for which many reports conclude the advantageous use of convolutional neural networks (CNNs) in prostate lesion detection and classification (PLDC). However, the network training inevitably involves prostate magnetic resonance (MR) images from multiple sites/cohorts. There always exists variation in scanning protocol among the cohorts, inducing significant changes in data distribution between source and target domains. This challenge has greatly limited clinical adoption on a large scale. Recent domain adaptation (DA) models can alleviate the inherent cross-site heterogeneity. Some could leverage cross-domain knowledge transfer using whole-slide images, without prior knowledge of lesion regions. In this paper, we propose a coarse mask-guided deep domain adaptation network (CMD²A-Net) in order to develop a fully automated framework for PLDC using multi-cohort images. No category or mask label is required from the target domain. A coarse segmentation module is trained to cover the possible lesion-related regions, so that attention maps can be generated to dedicate the local feature extraction of lesions within those regions. As a result, the features of both prostate lesion and region can be fused to align the robust features between the source and target domains. Experiments have been performed on 512 mpMRI sets from datasets of PROSTATEx (with 330 sets) and two cohorts, A (with 74 sets) and B (with 108 sets). Using the ensemble learning, our CMD²A-Net accomplishes an AUC of 0.921 in cohort A and 0.913 in cohort B, demonstrating its transferability from a large-scale public dataset PROSTATEx to our small-scale target domains. Our results and ablation study also support the CMD²A-Net’s effectiveness in lesion classification between benign or malignant, compared to the state-of-the-art models Corresponding author(s) Email:   kwokkw@hku.hk (K.W. Kwok), varv@hku.hk (V. Vardhanabhuti)

AbstractFabrication of structures in unstructured environments is a promising field to expand the application spaces of additive manufacturing (AM). One potential application is to add new components directly onto existing structures. In this paper, we developed a versatile, reconfigurable direct ink write (DIW) manufacturing method in tandem with a two-stage hybrid ink designed to fabricate high-strength, self-supporting parts in unconventional printing spaces, such as underneath a build surface or horizontally. Our two-stage hybrid DIW ink combines a photopolymer and a tough epoxy resin. The photopolymer can be cured rapidly to enable layer-by-layer printing complex structures. It also possesses adequate adhesion to allow the fabrication of large volume structures on a diversity of substrates including acrylic, wood, glass, aluminum, and concrete. The epoxy component can be cured after 72 hours in ambient conditions with further increased adhesion strengths. We demonstrated the capabilities of the reconfigurable DIW extrusion nozzle method to print complex structures in inverted and horizontal environments. Finally, via the addition of DIW-deposited conductive paths, we created a functional 3D printed structure capable of in-situ deformation monitoring. This work has the potential to be used for applications such as appending new parts to existing structures for increasing functionality, repair, and structure health monitoring.Corresponding author: H. Jerry Qi, qih@me.gatech.edu

The interest in soft pneumatic actuators has been growing rapidly in robotics, owing to the contact adaptability with the material softness. However, these actuators are mostly controlled by rigid electronic pneumatic valves, which can hardly be integrated into the robot itself, limiting its mobility and adaptability. Recent advances in soft or electronics-free valve designs provide the potential to achieve an integrated soft robotic system with reduced weight and rigidity. Nevertheless, the challenge in valve response remains open. To enable dynamic control of a soft pneumatic actuator, a fast-response proportional valve is needed. In this paper, we explored the potential of Ecoflex-based magnetorheological elastomer (MRE) membrane to create a proportional valve that can be used in the control of a soft robot made from the same silicone material. Experimental characterization shows that the proposed MRE valve (30 mm \(\times\) 30 mm \(\times\) 15 mm, 30 grams) can hold pressure up to 41.3 kPa and regulate the airflow in an analog manner. The valve is used to perform closed-loop Proportional-Integral-Differential (PID) control with 50 Hz on a soft pneumatic actuator and is able to control the pressure within the actuator chamber with a root-mean-square error of 0.05 kPa.   Corresponding author(s) Email:   sihan.wang@eng.ox.ac.uk  

AbstractMotile cilia move in an asymmetric pattern and implement a metachronal wave (MCW) to facilitate fluid movement in a viscous environment. Studies have been conducted to mimic MCW movement of motile cilia, but the fabrication process was too complicating or there were difficulties in accurately mimicking the shape of the cilia. To overcome these limitations, we introduce a self-assembly method to fabricate a reprogrammable magnetically actuated self-assembled (RMS) cilia array that can be reprogrammed by changing the magnetization direction through additional magnetization. Using the RMS cilia array, a unilateral cilia array (UCA) channel and a bilateral cilia array (BCA) channel were constructed, and the motion and fluid flow of the RMS cilia array were analyzed by applying different magnetic fields (strike magnetic field and rotating magnetic field). When a rotating magnetic field was applied to the UCA channel, a distinct MCW appeared. In the BCA channel test, fluid pumping was observed when a strike magnetic field, whereas fluid mixing was observed when a rotating magnetic field was applied. Based on these results, it is expected that the proposed RMS cilia array and magnetic field actuation method can be applied to lab-on-a-chip or microfluidic channels for fluid mixing and pumping.1. IntroductionCilia are hair-like, microtubule-based structures that have various distributions with a length of approximately 3–200 µm and an aspect ratio ranging from 10 to 100, depending on the location where they are found, and are divided into primary cilia and motile cilia.[1–5] Motile cilia can move objects or mix fluids by moving mucus or body fluids in the human body.[6,7] Among these motile cilia, the cilia that are found in the fallopian tube of the female reproductive system help the movement of the ovary, and the cilia existing in the lungs mix settled dust and bacteria through mucociliary clearance and move them out of the body.[8–10] The environment in which motile cilia move is normally filled with fluid with a low Reynolds number. In such an environment, the viscous force is generally more dominant than the inertial force, and has a significant influence on the fluid flow.[11–13] To be helpful in this environment, the cilium moves in an asymmetrical pattern comprising an effective stroke and a recovery stroke, creating a net fluid flow. In an effective stroke, the cilium moves in an arc that is fully stretched, while in the recovery stroke, the cilium returns to the starting point in a bent state as if swinging, which increases the moving area of the cilium.[14–16] In addition, when several cilia gather to form a cilia array, they move in a sequential pattern that forms a wave called the metachronal wave (MCW), which helps move the fluid faster and more efficiently because of their asymmetrical motion.[17–20]Several studies have reported mimicking the asymmetric motion of cilia and the MCW motion to efficiently pump or mix fluids in microfluidic devices with low Reynolds numbers.[21–23] To mimic cilia motion, many actuation methods have been used; actuation via a magnetic field is the most used method among them.[24–27] In addition, diverse manufacturing methods exist for magnetically actuated artificial cilia, and, the fabrication method using self-assembly has the advantages of simplicity and capability to mimic the appearance of natural cilia. However, it is difficult to program the magnetization direction, thus limiting the implementation of the MCW of the cilia.[28–31]Nevertheless, studies have been conducted to imitate the metachronal wave of cilia by fabricating artificial cilia using the molding method, which facilitates reprogramming.[32,33] Nelson formed a cilia array using a molding method and then reprogrammed the cilia array, which moved the cilia array to form an MCW.[34] Sitti fabricated micro-cilia using a mold, magnetized each cilium independently, and attached them to form a cilia array with the desired arrangement that implements the MCW.[35] However, these studies actuated the cilia array using only a rotating external magnetic field, and because the cilia array was fabricated using the molding method, several complex steps were required for making the cilia array.

More than 120 million mice and rats are used yearly for scientific purposes. While tracking their motion behaviors has been an essential issue for the past decade, present techniques, such as video-tracking and IMU-tracking have considerable problems, including requiring a complex setup or relatively large IMU modules that cause stress to the animals. Here, we introduce a wireless IoT motion sensor (i.e., weighing only 2 grams) that can be attached and carried by mice to collect motion data continuously for several days. We also introduce a combined segmentation method and an imbalanced learning process that are critical for enabling the recognition of common but random mouse behaviors (i.e., resting, walking, rearing, digging, eating, grooming, drinking water, and scratching) in cages with a macro-recall of 94.55%. Corresponding author(s) Email:  _ hoychan@cityu.edu.hk; xinyueli@cityu.edu.hk; wenjli@cityu.edu.hk  _

In existing surgery process, surgeons need to manually adjust the laparoscopes to provide a better field of view during operation, which may distract surgeons and slow down the surgery process. This paper presents a data-driven control method that uses a continuum laparoscope to adjust the field of view by tracking the surgical instruments. A Koopman-based system identification method is firstly applied to linearize the nonlinear system. Shifted Chebyshev polynomials are used to construct observation functions that transfer low-dimension observations to high-dimension ones. The Koopman operator is approximated using a finite-dimensional estimation method. An optimal controller is further developed according to the trained linear model. Furthermore, a learning-based pose estimation framework is designed to detect keypoints on surgical instruments and provide visual feedback for adjusting the laparoscope. Compared with other detection methods, the proposed scheme achieves a higher detection precision and provides more optional keypoints for tracking. Simulation and experiments validate the feasibility of the proposed control method. Experiment results show that the proposed method can automatically adjust the field of continuum laparoscope through tracking surgical instruments in a timely manner and the number of surgical tools is not limited.